Source-linked AI summary
History Repeats Itself: Human Motion Prediction via Motion Attention
Wei Mao, Miaomiao Liu, Mathieu Salzmann
TL;DR
Existing motion predictors struggle with long-term history and do not adequately emphasize historical subsequences that match the current motion context. This paper introduces motion attention with a graph-based predictor, achieving state-of-the-art performance across motion-prediction benchmarks and generalizing to unseen datasets without retraining or fine-tuning.
Problem
Existing recurrent predictors struggle to track long-term history, while long-term encoders treat all historical motion equally instead of emphasizing contextually relevant past segments.
Method
The approach compares the latest observed motion subsequence with historical subsequences, aggregates their future-motion representations through motion attention, and uses the result for prediction.
Results
The approach achieves state-of-the-art performance on commonly used motion-prediction benchmarks and generalizes to previously unseen datasets without retraining or fine-tuning.
Takeaways & Limitations
Selective use of distant historical motions supports prediction across short- and long-term horizons, including non-periodical motions with repeated history.
Takeaways & Limitations
The attention construction assumes each historical subsequence contains M past frames followed by T future frames for prediction.
Abstract
from arXiv · showhide
Human motion prediction aims to forecast future human poses given a past motion. Whether based on recurrent or feed-forward neural networks, existing methods fail to model the observation that human motion tends to repeat itself, even for complex sports actions and cooking activities. Here, we introduce an attention-based feed-forward network that explicitly leverages this observation. In particular, instead of modeling frame-wise attention via pose similarity, we propose to extract motion attention to capture the similarity between the current motion context and the historical motion sub-sequences. Aggregating the relevant past motions and processing the result with a graph convolutional network allows us to effectively exploit motion patterns from the long-term history to predict the future poses. Our experiments on Human3.6M, AMASS and 3DPW evidence the benefits of our approach for both periodical and non-periodical actions. Thanks to our attention model, it yields state-of-the-art results on all three datasets. Our code is available at https://github.com/wei-mao-2019/HisRepItself.
1 Introduction
The paper addresses long-term motion prediction by dynamically retrieving historically similar motion sub-sequences rather than treating all past frames equally. Its motion-attention predictor exploits repeated patterns across periodic and complex activities and achieves strong performance across datasets.
- Problem: Deep models outperform traditional methods on complex motions, but RNNs often generate static poses at mid- to long-term horizons.Existing long-term encoders also treat the entire motion history equally.
- Motivation: Humans repeat motion in both short periodic activities such as walking and longer complex actions such as sports and cooking.The method therefore searches historical motion for contextually relevant information.
- Motivation: Frame-wise pose attention can confuse similar static poses from different motions because it does not capture motion direction.The proposed alternative compares the latest visible sub-sequence with historical motion sub-sequences.
- Approach: The approach uses DCT-encoded motion history, attention-weighted aggregation, the latest observed motion, and a graph convolutional network for spatial dependencies.The aggregated history and latest motion are combined to estimate future motion.
2 Related Work
Prior work uses recurrent models, manually configured temporal encoders, or frame-wise pose attention for motion prediction. This paper instead uses motion-subsequence attention with DCT representations and a feed-forward prediction network.
- RNN-based human motion prediction: RNN-based methods have been widely used for human motion prediction, including encoder-recurrent-decoder architectures and autoregressive predictors.These methods address sequence modeling but remain part of the recurrent baseline family.
- Long-term temporal modeling: A prior long-term approach extracts a motion-history prior, but its manually selected convolutional-window size strongly affects temporal encoding.This introduces sensitivity to the chosen temporal window.
- DCT and graph convolution: DCT-based short-term encoding with a graph convolutional network previously achieved state-of-the-art performance up to 1s, while long-term history encoding remained problematic.The present work builds on this representation while addressing historical-motion selection.
- Attention-based prediction: Frame-wise pose attention can produce ambiguous motion because static poses omit direction and recur in substantially different motions.Motion attention compares sub-sequences rather than individual poses.
- Attention-based prediction: Combining motion attention with a feed-forward prediction network enables the method to outperform state-of-the-art motion-prediction frameworks.The comparison is reported as experimental evidence for the proposed design.
3 Our Approach
The approach matches the latest motion context to historical sub-sequences, aggregates relevant future-motion representations, and combines them with recent observations in a GCN predictor. It uses DCT-based temporal representations and achieves strong short- and long-term results on H3.6M.
- Motion Attention: Motion attention compares the latest sub-sequence with historical sub-sequences rather than static poses to exploit repeated motion patterns.The model represents the latest sub-sequence as a query and historical sub-sequences as key-value pairs, using partial motion similarity for aggregation.
- Motion Attention: The motion history is divided into overlapping sub-sequences of M + T poses, with the first M poses serving as keys and the full sub-sequences as values.The latest M-pose sub-sequence is used as the query, while each value contains both observed and future motion.
- Motion Attention: Attention scores weight DCT-encoded historical motions, whose aggregated output is combined with the latest observed motion to estimate future motion.High-frequency DCT components may be truncated to avoid predicting jittery motion, and the estimate is processed by the prediction network.
- Experiments: Our approach achieves state-of-the-art short-term performance across 15 H3.6M actions at almost all time horizons, especially for repetitive actions such as Walking.The short-term table measures 3D joint-position error in millimeters.
- Prediction Model: The predictor concatenates the padded recent-motion DCT coefficients with the attention output and uses graph convolutions to model dependencies among joint coordinates or angles.The human body is represented as a fully connected graph with trainable adjacency matrices and feature-extraction weights.
- Experiments: On average, our approach performs best for long-term 3D joint-position prediction on H3.6M.The reported long-term results are summarized in Table 2.
4 Experiments
Experiments evaluate the method on Human3.6M, AMASS, and 3DPW using short- and long-term prediction settings. The results show strong performance, generalization, and improved use of repeated motion in distant history.
- Datasets and evaluation: The evaluation covers Human3.6M, AMASS, and 3DPW, with short-term and long-term prediction reported in joint-angle and 3D-position settings.3DPW is evaluated using a model trained on AMASS to assess generalization.
- Human3.6M results: On Human3.6M, the method outperforms all baselines on average for both short-term and long-term 3D prediction.Improvements are larger for actions with clearly repeated history, including Walking and Walking Together.
- Human3.6M results: The method achieves state-of-the-art performance for both short-term and long-term prediction, unlike LTD variants that trade performance across horizons.LTD-10-10 reaches 114.0mm average error at 1000ms, while LTD-10-25 and LTD-50-25 favor other horizons.
- Cross-dataset results: Across AMASS and 3DPW, the method consistently outperforms baseline approaches and generalizes better to the unseen 3DPW dataset.None of the methods were trained on 3DPW.
- Long-range history: Using 100 observed frames instead of 50 produces a 4.2mm performance boost at 1s when similar motion appears farther in the history.The benefit is small before 160ms but becomes clear at later horizons.
- Long-range history: When the past 40 frames are replaced by a constant pose, attending to similar motion between frames −80 and −60 yields a trajectory closer to ground truth than using only the past 50 frames.This tests whether distant historical motion contributes useful predictive information.
5 Conclusion
The paper concludes that motion attention selectively retrieves relevant historical sub-sequences, including motions far in the past. The approach achieves state-of-the-art benchmark performance, generalizes without retraining or fine-tuning, and benefits longer-history prediction.
- 5 Conclusion: Motion attention selects historical information according to similarity between the current motion context and past sub-sequences.The model can use relevant historical motions even when they are far in the past.
- 5 Conclusion: The approach achieves state-of-the-art performance on commonly used motion-prediction benchmarks and recently published datasets.The conclusion reports this result across the paper’s evaluated datasets.
- 5 Conclusion: The network generalizes to previously unseen datasets without retraining or fine-tuning and can use longer history than during training.Longer history further boosts performance on non-periodical motions with repeated history.
1 Datasets
The experiments use Human3.6M, AMASS, and 3DPW, with dataset-specific preprocessing and splits for motion-prediction evaluation.
- Human3.6M: Human3.6M is represented with a standard skeleton and processed into 48-dimensional angle or 66-dimensional 3D-position pose vectors.Global rotation, translation, and constant angles or coordinates are removed.
2 Implementation Details
The implementation trains the network with Adam using scheduled learning-rate decay, fixed epoch counts, and dataset-specific batch sizes.
- Implementation: The model uses Adam with an initial learning rate of 0.0005, decayed to 0.00005 by epoch 50, and trains for 50 epochs.Batch sizes are 32 for Human3.6M and 128 for AMASS.
- Implementation: The reported implementation processes one forward and backward pass in 32ms on Human3.6M and 45ms on AMASS using an NVIDIA Titan V GPU.
3 Additional Results on H3.6M
On Human3.6M, the approach achieves state-of-the-art performance for both short- and long-term prediction, while generating highly realistic long futures for periodic motions.
- The approach achieves state-of-the-art performance on average for both short-term and long-term Human3.6M prediction.Results average errors over 256 random sub-sequences per action.
- For periodic “Walking” motions, the method generates futures up to 16 seconds that humans find hard to distinguish from ground truth.
4 Additional Results on AMASS
On BMLrub, the approach produces predictions that better match the ground truth than LTD, particularly for the legs.
- The approach better matches the ground truth than LTD on the BMLrub dataset.The qualitative comparison uses 3D position predictions.
- The qualitative advantage is particularly visible in the predicted legs.
5 Motion Attention vs. Frame-wise Attention
Motion attention outperforms frame-wise attention because it compares motion sub-sequences rather than isolated poses, thereby accounting for motion direction.
- Motion attention assigns attention using historical motion sub-sequences and the latest observed motion context.The model uses the first M poses of each historical sub-sequence as keys and the last observed M poses as the query.
- Frame-wise Attention instead uses the last frame of each historical sub-sequence as keys and the final observed frame as the query.
- Motion attention outperforms frame-wise attention by a large margin in Table 3.The comparison is presented in the table caption as a comparison between frame-wise attention and motion attention.
- Frame-wise attention creates ambiguities because it does not consider the direction of motion.