Source-linked AI summary
Context-aware Human Motion Prediction
Enric Corona, Albert Pumarola, Guillem Alenyà, Francesc Moreno-Noguer
TL;DR
Human motion prediction traditionally relies on past 3D skeletons while omitting environmental influences. The paper introduces a semantic-graph architecture that learns interactions from human and object observations and uses them to predict future motion, with context-aware models outperforming context-free baselines across the reported evaluations.
Problem
Standard human-motion prediction omits how objects and other people influence and constrain human movement, despite this context being relevant to motion forecasting.
Method
A directed semantic graph represents humans and objects as nodes, iteratively refines their contextual states from past observations, and supplies context representations to motion-prediction RNNs.
Results
Context-aware models clearly outperform baselines without context while improving both human and object motion prediction across the evaluated actions and datasets.
Takeaways & Limitations
Modeling scene context yields predictions that are more semantically related to nearby objects and supports coherent human-object motion forecasting.
Abstract
from arXiv · showhide
The problem of predicting human motion given a sequence of past observations is at the core of many applications in robotics and computer vision. Current state-of-the-art formulate this problem as a sequence-to-sequence task, in which a historical of 3D skeletons feeds a Recurrent Neural Network (RNN) that predicts future movements, typically in the order of 1 to 2 seconds. However, one aspect that has been obviated so far, is the fact that human motion is inherently driven by interactions with objects and/or other humans in the environment. In this paper, we explore this scenario using a novel context-aware motion prediction architecture. We use a semantic-graph model where the nodes parameterize the human and objects in the scene and the edges their mutual interactions. These interactions are iteratively learned through a graph attention layer, fed with the past observations, which now include both object and human body motions. Once this semantic graph is learned, we inject it to a standard RNN to predict future movements of the human/s and object/s. We consider two variants of our architecture, either freezing the contextual interactions in the future of updating them. A thorough evaluation in the "Whole-Body Human Motion Database" shows that in both cases, our context-aware networks clearly outperform baselines in which the context information is not considered.
1. Introduction
Human motion forecasting is important for interaction with people and environments, but standard sequence-to-sequence approaches typically use only past 3D skeletons. This paper instead models environmental interactions and evaluates context-aware prediction on datasets containing people and objects.
- Human motion prediction supports applications including human-robot interaction, autonomous driving, motion generation, and image understanding.
- Recent methods typically feed past 3D skeleton observations into an RNN to predict near-future movements, usually under two seconds.
- Standard formulations omit how objects or other people influence and constrain human movement.
- The proposed architecture represents people and scene objects as nodes in a directed semantic graph whose edges encode learned mutual interactions.
- The approach is evaluated on the Whole-Body Human Motion Database and CMU MoCap, with context-aware versions reported to outperform context-free baselines.
2. Related work
Prior human-motion methods largely predict future poses from historical human motion without context, while related work separately studies object motion, human-object relationships, and graph-based reasoning. This paper positions context-aware motion prediction as addressing that gap.
- Human motion prediction: Most human-motion approaches use RNNs to encode historical human movement and predict future configurations, with some newer methods adding adversarial or geometric losses.
- Human motion prediction: Alternative human-motion models use skeleton-shaped spatio-temporal graphs or convolutional networks instead of standard RNNs.
- Human motion prediction: Existing human-motion formulations generally exclude context information, which this paper aims to incorporate.
- Rigid 3D object motion prediction: Rigid-object forecasting has received relatively little attention, and prior approaches may use actions explicitly whereas this paper encodes actions implicitly in previous observations.
- Human-Object Interaction: Human-object interaction research commonly represents detected people and objects with graphs or other structured representations, but this paper focuses on motion prediction rather than relationship identification.
- Graph-based context reasoning: Related graph-based methods use context reasoning for tasks such as classification, spatio-temporal modeling, and image segmentation.
3. Problem formulation
The paper reformulates human motion forecasting so future human and object motion depends on past human motion, object observations, and task information, while also predicting future interactions. This extends the conventional mapping from past poses to future poses.
- Conventional forecasting maps past human poses P_t0:t−1 to future poses P_t:tf.
- The paper hypothesizes that future human motion is also driven by scene context, including objects and their interactions with the person.
- The expanded model takes past human poses, object observations, and task information as input, and predicts future human motion, object motion, and interactions.
- The model also aims to learn how objects influence one another’s future motion in an unsupervised manner.
4. Approach
The approach represents people and objects as nodes in a directed semantic graph, iteratively refining their context states and interactions from observed motion. It combines these representations with recurrent prediction to forecast human and object movements, either holding contextual interactions fixed or updating them into the future.
- Architecture: The architecture separately processes human motion and object relationships, then uses the resulting entity representation to predict human and object motion.The human branch builds on a residual RNN that predicts body-joint velocities, while the context branch models object relationships.
- Entity representation: Each graph node represents a person or object with a state containing its 3D bounding box, object type, and human joints when applicable.Non-person nodes use zero vectors for joints, while the one-hot object type helps identify the task and associated motion.
- Context representation: The context graph uses normalized weighted adjacency matrices to represent interactions among nodes, with node updates aggregating information through graph convolutions or attention-like operations.The normalized adjacency matrix defines node interactions; the graph-convolution formulation combines neighboring representations using trainable weights and an activation function.
- Context representation: A second RNN maintains hidden context representations for every scene node and updates them at each observed frame.These representations encode context relevant to each entity and are iteratively refined as observations arrive.
- Learning interactions: Interactions are initially set to self-dependencies, then estimated from pairs of node hidden states as asymmetric weighted adjacency matrices and normalized with Softmax.The model learns changing interactions without supervision; a simpler evaluated variant instead connects objects whose centers of gravity are within 1 meter.
- Future prediction: The model has two context-aware variants: one injects refined past context into human prediction, while the other also predicts object motion and propagates context into future steps.The future-updating variant forwards predicted object positions and updates person-node joint features using human predictions; multi-person tracking repeats the human branch for each person.
5. Implementation details
The implementation uses residual RNNs, graph-based object representations, fixed observation and prediction windows, and L2 training. Table 1 evaluates class-specific models, while Table 2 trains one model across all actions.
- The human and object RNN hidden representations contain 1024 and 256 dimensions, respectively.
- Models encode 10 frames over 1 second and decode 20 frames over 2 seconds, sampling observations every 100 ms.Longer encoding windows did not improve results and increased training time.
- The adjacency matrix uses pairwise concatenated hidden representations, while two 1×1 convolutional layers form the interaction function.The convolutional layers and edge convolutions use no bias term.
- Table 1 reports class-specific mean Euclidean error for 2-second human and object predictions, while Table 2 trains a single model across all actions.Both settings provide 1 second of past observations; the context-based OMP and LI models achieve the best results in most sequences in Table 1.
- Object representations begin with bounding-box positions defined by minimum and maximum 3D Cartesian points.
- Training minimizes L2 distance between predicted and actual future motion using Adam with learning rate 0.0005 and batch size 16.The model is trained until convergence with beta1 0.5 and beta2 0.99.
6. Experiments
Experiments evaluate context-aware motion prediction on datasets containing human-object or human-human interactions, using progressively richer context-aware models and qualitative interaction analyses. Results show stronger gains when interactions are well defined, with improvements over context-less baselines in motion prediction, interaction modeling, CMU evaluation, and noisy-input settings.
- Experimental setup: The WBHM experiments use 190 videos, 198K frames, and 15 tracked object classes, with 18-joint human skeletons extracted from Vicon recordings.The selected activities include human joints and at least one table.
- Experimental setup: The models are compared with context-less RNN, Zero-Velocity, and QuaterNet baselines, while C-RNN variants incrementally add context processing, object motion prediction, and learned interactions.The basic C-RNN uses distance-based interactions; extensions predict object motion and recompute interactions from predicted positions.
- WBHM results: Learned-interaction models substantially improve most cases, whereas threshold-based binary interactions vary across classes and basic C-RNN does not consistently improve over state-of-the-art models.Passing objects and grasping remain challenging because relevant items may occur at variable distances.
- WBHM results: Most improvement occurs in scenes with few objects and well-defined interactions; cluttered cooking scenes reduce the advantage of context-aware models because motion options and uncertainty increase.Training across all actions favors context-aware models, especially C-RNN+LI and C-RNN+OMP+LI.
- Qualitative results: Context-aware predictions are often more semantically related to nearby objects, while context-less object predictions remain close to their initial positions.One example is people holding objects moving the relevant hand.
- CMU results: On CMU MoCap, C-RNN+LI significantly outperforms all other baselines, especially in the long term, despite tracking only two people and having no annotated objects.CMU errors average more than twice those on the other database because the activities involve larger absolute motion.
- Robustness to noise: With noisy observations, prediction error increases gracefully, yet C-RNN+LI and C-RNN+OMP+LI remain better than context-less baselines even with noise up to σ = 50mm.The evaluation corrupts past human and object coordinates with additive Gaussian noise.
7. Conclusion
The proposed context-aware models use semantic graphs to represent humans and objects, and improve human and object motion prediction over state-of-the-art models without context.
- Semantic-graph models represent humans and objects as nodes, supporting environments with varying numbers of objects and complexity.
- Across different actions, the proposed models understand human activities significantly better than state-of-the-art models that do not use context.
- Context-aware modeling improves prediction for both human and object motion.