Source-linked AI summary
Long-term Human Motion Prediction with Scene Context
Zhe Cao, Hang Gao, Karttikeya Mangalam, Qi-Zhi Cai, Minh Vo, Jitendra Malik
TL;DR
Long-term human motion prediction needs to account for goal-directed behavior and scene layout, which existing methods largely neglect. The paper introduces a three-stage, scene-aware framework and a synthetic dataset with clean annotations, reporting quantitative and qualitative improvements over existing methods.
Problem
Existing human-motion prediction methods do not use scene context, limiting long-term prediction of goal-directed poses and locations.
Method
Given a scene image and 2D pose history, the framework samples motion goals, plans 3D paths, and predicts 3D pose sequences, using a synthetic dataset for training.
Results
The method outperforms existing methods quantitatively and qualitatively on synthetic and real datasets.
Takeaways & Limitations
Scene context and goal-oriented factorization provide a framework for predicting long-term stochastic 3D human motion from 2D inputs.
Takeaways & Limitations
Without reconstructed 3D meshes and offline optimization, predicted poses may not satisfy all physical scene-geometry constraints.
Abstract
from arXiv · showhide
Human movement is goal-directed and influenced by the spatial layout of the objects in the scene. To plan future human motion, it is crucial to perceive the environment -- imagine how hard it is to navigate a new room with lights off. Existing works on predicting human motion do not pay attention to the scene context and thus struggle in long-term prediction. In this work, we propose a novel three-stage framework that exploits scene context to tackle this task. Given a single scene image and 2D pose histories, our method first samples multiple human motion goals, then plans 3D human paths towards each goal, and finally predicts 3D human pose sequences following each path. For stable training and rigorous evaluation, we contribute a diverse synthetic dataset with clean annotations. In both synthetic and real datasets, our method shows consistent quantitative and qualitative improvements over existing methods.
1 Introduction
The paper frames long-term 3D human motion prediction as goal-directed behavior conditioned on scene context, then addresses it with a three-stage framework and a synthetic dataset with clean annotations.
- The task predicts future human poses and locations from a scene image and a person’s past 2D pose and location history.
- Scene layout influences goals, collision-avoiding paths, and pose sequences, motivating spatially contextualized motion prediction.
- Existing pose-prediction frameworks ignore scene context and therefore tend toward short-term, local motion such as walking in place.
- The proposed framework samples multiple 2D destinations, predicts a 3D path for each, and generates 3D pose sequences along those paths.
- The authors contribute a large-scale synthetic human-scene interaction dataset with over one million HD RGB-D frames and clean 3D annotations.Pre-training on this dataset stabilizes training and improves prediction performance on a real dataset.
- The method is reported to outperform existing methods quantitatively and qualitatively while formulating long-term 3D motion prediction with scene context.
2 Related Work
Prior work separately studies global trajectory prediction and local pose prediction, while scene-aware approaches rely on restricted viewpoints, known 3D information, or expensive optimization.
- Global trajectory prediction: Earlier trajectory methods model social-scene interactions using physical forces, continuum dynamics, Hidden Markov models, or game theory.
- Global trajectory prediction: Neural trajectory predictors capture multimodal scene-agent interactions but commonly assume bird’s-eye-view images or known 3D information.
- Local pose prediction: Neural local pose predictors include recurrent, temporal convolutional, and frequency-space graph convolutional networks, but ignore image context.
- Local pose prediction: Ignoring scene context can produce motion inconsistent with the environment, such as walking through walls.
- Scene-aware prediction: Before this work, no learning-based method holistically modeled scene context and human pose for more than a single time instance.
3 Approach
The approach factorizes long-term 3D motion prediction into goal sampling, scene-aware 3D path planning, and pose generation following each path. Its modular design supports stochastic or deterministic predictions while using scene images and 2D pose histories as inputs.
- Overall framework: The model predicts future 3D poses and locations from an N-step 2D pose history, a scene image, and a known camera model.The human skeleton is represented with J keypoints, with 2D inputs and 3D outputs.
- GoalNet: Predicting 2D Path Destination: GoalNet samples multiple plausible 2D motion destinations conditioned on the pose history and scene image.It uses a conditional variational auto-encoder and samples latent variables to represent uncertainty.
- Prediction modes: The modular system supports stochastic prediction by sampling destinations and deterministic prediction by removing GoalNet and directly using PathNet.Both modes are evaluated in the experiments.
- Training: Training uses destination, path, and pose objectives, including destination error, KL divergence, path supervision, smoothness, and 3D pose error.Path losses include 2D heatmap and 3D coordinate supervision, with a penalty for large changes between consecutive frames.
- PoseNet: Generating 3D Pose following Path: PoseNet predicts 3D pose sequences following each predicted path, refining an initial 3D pose estimate obtained by lifting 2D history and replicating the present pose along future path locations.During training, coarse pose inputs use ground-truth paths; during testing, they use predicted paths.
4 GTA Indoor Motion Dataset
GTA-IM is a synthetic indoor human-motion dataset designed for human-scene interaction, addressing noisy annotations and limited motion range in existing real datasets. It provides diverse scenes, activities, viewpoints, and clean 3D annotations at large scale.
- Dataset design: GTA-IM targets human-scene interaction with automatically controlled characters, cameras, action tasks, lighting, and motion conditions.The collection randomizes destinations, tasks, walking styles, movement speeds, weather, daytime, camera locations, and viewing angles.
- Motivation and coverage: The dataset was motivated by limitations of existing real and synthetic datasets, including noisy 3D annotations, depth-sensor-limited motion, contrived environments, and limited interactable objects.The collection includes activities such as climbing stairs, lying down, sitting, and opening doors.
- Dataset scale and annotations: The dataset contains one million 1920 × 1080 RGBD frames with ground-truth 3D human poses, segmentation maps, and camera poses.The pose annotations contain 98 joints.
- Dataset scale and annotations: GTA-IM includes 50 human characters acting in 10 large indoor scenes with multiple floors and room types.Scenes include living rooms, bedrooms, kitchens, and balconies, enabling diverse interaction activities.
5 Evaluation
The evaluation spans synthetic GTA-IM and real PROX data, comparing deterministic and stochastic scene-aware predictions with quantitative and qualitative baselines. Results show improved accuracy, stronger long-term performance from sampling, more plausible scene-consistent motion, and a remaining physical-constraint limitation.
- Datasets and metrics: Evaluations use GTA-IM and PROX, measuring 3D path and pose prediction with mean MPJPE across time steps.GTA-IM uses eight training and two evaluation scenes; PROX uses 52 training and eight testing sequences.
- Quantitative comparison: 173 mm versus 193 mm: the deterministic image-input model outperforms other GTA-IM methods, while ten stochastic samples reduce error to 165 mm.Four samples match the deterministic model at 173 mm, and ten samples further lower the reported error.
- Ablations: Image context and the proposed path representation matter: removing images raises average error by 23 mm, while direct xyz regression is 18 mm worse.Masked-image input is 16 mm worse than the full model; the authors attribute the path representation advantage to its correlation with image appearance.
- Quantitative comparison: 270 mm versus 282 mm: the method outperforms prior state of the art on PROX, while ten samples reduce its error to 249 mm.Three samples already beat the deterministic model, at 264 mm versus 270 mm.
- Qualitative and long-term results: Qualitative results show fewer scene collisions and more plausible motions, while destination sampling produces diverse activities and stochastic prediction improves long-term accuracy.At three seconds, stochastic predictions differ from deterministic results by more than 100 mm; examples include turning, stairs, standing, and reclining.
- Failure cases: The learned scene constraints do not guarantee strict physical validity: small foot intersections with floors or beds remain without reconstructed 3D meshes and explicit geometry refinement.The paper suggests multi-view or temporal images and explicit scene constraints as future remedies.
6 Conclusion
The paper presents long-term stochastic 3D human-motion prediction from 2D inputs while incorporating scene context, supported by a new dataset and method. It also identifies evaluation of naturalness, dynamic objects, and multiple moving people as future directions.
- The work studies long-term 3D human-motion prediction from only 2D input while considering the surrounding scene.
- The proposed method predicts stochastic future motion using scene context, and the paper contributes diverse human-scene recordings with clean 3D annotations.
- Future work includes evaluating the naturalness and feasibility of stochastic predictions and incorporating dynamic objects and multiple moving people.
- Qualitative examples show multiple plausible futures generated by sampling different movement goals, including turning, continuing, or retrieving objects.
A Dataset Comparison
Existing 3D human-motion datasets differ in environment diversity, annotation quality, scale, and viewpoint coverage. The proposed synthetic dataset addresses these limitations through an automatic gaming-engine pipeline with diverse recordings and clean annotations.
- Human3.6M provides accurate 3D poses but mainly records people in laboratory environments with limited human interaction with indoor surroundings.
- PiGraphs and PROX model human-scene interaction but contain relatively few frames, scenes, and characters, with limited camera and background diversity.
- The proposed dataset uses an automatic gaming-engine pipeline to collect large-scale, diverse recordings with clean annotations.
B Implementation Details
The implementation trains the three modules separately rather than jointly and uses distinct training configurations for the datasets and PoseNet.
- All modules are implemented in PyTorch and optimized with ADAM.
- The three modules are trained separately because separate training performs better than joint training for this multimodal problem.
- Training schedules differ across GTA-IM, PROX, and PoseNet, including separate epoch counts, learning rates, decay points, dropout, and batch sizes.
C GoalNet Evaluation
GoalNet is evaluated for stochastic 2D future-destination prediction on GTA-IM and PROX, while long-term path and pose errors are compared across sample counts.
- GoalNet directly predicts 2D destinations and is compared with deterministic PathNet using least and average error among samples.
- With 5 samples, GoalNet achieves performance similar to the deterministic baseline on both GTA-IM and PROX.
- With 30 samples, GoalNet reduces error by around 40% on GTA-IM and 20% on PROX relative to the deterministic baseline.
- Long-term evaluations compare stochastic and deterministic 3D path and pose error curves across prediction timesteps and sample counts.
D Long-term Evaluation on PROX
On PROX, the authors evaluate long-term 3D path and pose prediction with deterministic and stochastic models, finding that stochastic prediction surpasses deterministic prediction with three samples.
- On PROX, the evaluation covers predicted 3D paths and predicted 3D poses.
- The comparison includes deterministic prediction and stochastic prediction with varying numbers of samples.
- 3 samples let the stochastic model beat its deterministic counterpart on PROX, compared with 5 samples on GTA-IM.
E Qualitative Results
The paper presents additional qualitative comparisons of baseline and proposed 3D path and pose predictions through visualizations and video results.
- Figure 11 provides additional qualitative comparison results.
- The paper directs readers to a video for more results on 3D path and 3D pose prediction.
- The video is available through the project website.
F Network Architecture
The network architecture section specifies GoalNet and PathNet, while PoseNet uses a modified architecture based on prior work.
- The paper defines GoalNet and PathNet architectures in Tables 5 and 6, respectively.
- PoseNet follows prior work with its input embedding, output embedding, positional encoding, and softmax layers removed.
- Table 5 describes GoalNet using convolutional blocks with skip connections, batch normalization, and ReLU layers.
- Table 6 describes PathNet using convolutional and linear operators with normalization and ReLU layers, except before predictions.
- Table 7 documents the modular architecture for one HourglassStack, following the design and implementation of prior work [28].