Source-linked AI summary
Learning Human Motion Models for Long-term Predictions
Partha Ghosh, Jie Song, Emre Aksan, Otmar Hilliges
TL;DR
Long-horizon human-motion prediction requires models that capture both spatial skeleton structure and temporal dynamics while supporting evaluation without ground-truth sequences. The paper proposes DAE-LSTM, which learns these components from data and filters recurrent predictions, and reports improved prediction accuracy on two public datasets alongside a long-horizon naturalness protocol.
Problem
Long-horizon human-motion prediction is challenging because natural motion is stochastic and context-dependent, while generated sequences need evaluation beyond ground-truth distance measures.
Method
DAE-LSTM combines a 3-layer LSTM for temporal modeling with a dropout autoencoder that learns skeletal structure by reconstructing randomly removed joints and filters predicted poses.
Results
The model improves prediction accuracy over the state of the art on two publicly available datasets and supports naturalness analysis over long horizons.
Takeaways & Limitations
The proposed architecture and classifier-based protocol provide a data-driven approach for synthesizing and evaluating human motion over extended time horizons.
Takeaways & Limitations
The model lacks physics-based feedback and therefore has no concept of mass, balance, or gravity; the evaluation also excludes confusing discussion and smoking actions.
Abstract
from arXiv · showhide
We propose a new architecture for the learning of predictive spatio-temporal motion models from data alone. Our approach, dubbed the Dropout Autoencoder LSTM, is capable of synthesizing natural looking motion sequences over long time horizons without catastrophic drift or motion degradation. The model consists of two components, a 3-layer recurrent neural network to model temporal aspects and a novel auto-encoder that is trained to implicitly recover the spatial structure of the human skeleton via randomly removing information about joints during training time. This Dropout Autoencoder (D-AE) is then used to filter each predicted pose of the LSTM, reducing accumulation of error and hence drift over time. Furthermore, we propose new evaluation protocols to assess the quality of synthetic motion sequences even for which no ground truth data exists. The proposed protocols can be used to assess generated sequences of arbitrary length. Finally, we evaluate our proposed method on two of the largest motion-capture datasets available to date and show that our model outperforms the state-of-the-art on a variety of actions, including cyclic and acyclic motion, and that it can produce natural looking sequences over longer time horizons than previous methods.
1. Introduction
The paper addresses long-horizon human-motion prediction by learning spatial skeleton structure and temporal dynamics directly from data. Its DAE-LSTM architecture and long-horizon evaluation protocol target drift, motion degradation, and naturalness assessment.
- Motivation: Long-horizon human-motion prediction is difficult because natural motion is stochastic and context-dependent, yet applications require timely forecasts.Relevant applications include human-computer interaction, tracking, activity recognition, robotics, and image-based pose estimation.
- Approach: DAE-LSTM learns skeletal structure and temporal coherence directly from data without expert-designed, task-dependent spatio-temporal graphs.The model explicitly separates pose constraints from temporal constraints using a feed-forward pose filter and an RNN.
- Approach: Randomly removing entire joints during autoencoder training forces reconstruction from inter-joint dependencies, implicitly recovering the skeleton’s spatial configuration.The autoencoder operates directly in the human skeleton’s joint-angle domain rather than learning a lower-dimensional latent representation.
- Approach: The model predicts the next pose from prior poses, using recurrence to synthesize novel motion sequences while separating pose reconstruction from temporal modeling.The authors report improved performance over settings that primarily use the autoencoder for representation learning.
- Evaluation: A classifier-based protocol evaluates the naturalness of generated sequences over arbitrarily long horizons without requiring ground-truth sequences.The longer a sequence remains classified as the seed action, the higher the intended prediction quality.
- Results: On H3.6m and Holden et al.’s dataset, the model outperforms a 3-layer LSTM baseline and two state-of-the-art models for short- and long-horizon prediction.The proposed evaluation protocol is also used to analyze the generative motion task.
2. Related Work
Prior work uses spatio-temporal modeling, recurrent methods, and autoencoders for human-motion and activity tasks, but often relies on manual structure or task-specific models. This paper learns skeletal structure from data and combines it with an LSTM in a unified model.
- Spatio-temporal modeling: Spatio-temporal models support activity recognition, human-object interaction, robotics, and related tasks, but manually designed graphs require expertise and domain knowledge.Graph nodes represent interaction components, while edges capture their spatio-temporal relationships.
- Temporal modeling: Deep-learning approaches, especially recurrent neural networks, provide an alternative to explicit parametric filtering methods such as Gaussian processes, HMMs, and Kalman filters.The related work frames these as two broad directions for modeling temporal dependencies and state transitions.
- Autoencoders and motion prediction: Earlier motion-prediction methods use autoencoders for pose representation learning and RNNs for temporal variation, whereas this work operates directly in joint-angle space.The paper argues that mocap pose data is already smooth and continuous, without guarantees that learned latent spaces preserve those properties.
- Proposed model: The architecture combines a DAE with a 3-layer LSTM: the LSTM predicts future skeletal configurations and the DAE filters its noisy predictions at inference.This stacking is intended to reduce accumulated error and pose drift over time.
- Motion-prediction scope: Short-term and long-term accuracy remain challenging in prior work, with one cited approach reporting only up to 400 ms into the future.The paper positions longer-horizon prediction as important for tasks beyond articulated-object tracking.
- Proposed model: The proposed dropout autoencoder learns inter-joint dependencies by randomly removing joints, while an LSTM captures temporal evolution from reconstructed and denoised poses.Unlike related approaches, the model is unified, performs all actions, and does not require task-specific networks or expert-designed action-specific graphs.
3. Method
DAE-LSTM separates temporal prediction from spatial pose reconstruction: a 3-layer LSTM predicts future poses, while a dropout autoencoder learns skeletal structure and filters predictions. The components are pretrained independently, then fine-tuned together to improve long-horizon motion synthesis.
- Architecture: The architecture combines a 3-layer LSTM for temporal modeling with a Dropout Autoencoder that filters each predicted pose using learned skeletal constraints.The LSTM predicts Xt+1 from X1:t, and the autoencoder refines the result based on the human skeleton’s implicit spatial structure.
- Design rationale: The model explicitly separates spatial skeleton constraints from temporal coherence, avoiding expert-designed, task-dependent spatio-temporal graphs.The autoencoder models skeletal configuration, while the recurrent network models temporal aspects of motion.
- Training and prediction: The LSTM and DAE are pretrained independently, stacked, and briefly fine-tuned end-to-end; dropout is annealed to zero during fine-tuning.The fine-tuning phase lasts approximately 2 epochs and uses both component losses.
- Learning spatial structure: The Dropout Autoencoder randomly removes entire joints during training, forcing reconstruction from spatial correlations among adjacent joints.Unlike representation-learning autoencoders, it operates directly in the joint-angle domain and learns to recover missing joint information.
- Training and prediction: The DAE is pretrained without temporal information, while the LSTM can support deterministic or probabilistic pose prediction.The authors select deterministic prediction because few-component mixture models are inadequate for high-dimensional human poses and effectively collapse toward unimodal prediction.
- Inference and results: Filtering recurrent predictions counteracts assumed accumulated noise, and experiments report better sequence predictions across varied actions and longer locomotion horizons.The reported improvements include more realistic locomotion over longer horizons than the state of the art.
4. Experiments
The experiments evaluate DAE-LSTM on two motion-capture datasets, test its filtering and prediction accuracy, and assess long-horizon sequence naturalness with a classifier. Results show improved robustness to noise and strong performance across short- and long-term settings, with limitations in comparison coverage and some non-cyclic motions.
- Impact of the Dropout Autoencoder: DAE reconstruction remains more robust as input corruption increases, recovering noisy poses with lower error than Gaussian-noise and vanilla autoencoders.The comparison includes the proposed DAE, a Gaussian denoising autoencoder, and a vanilla autoencoder.
- Impact of the Dropout Autoencoder: DAE-LSTM improves filtered predictions by removing noise introduced by the recurrent model, and filtering at every timestep reduces long-term drift.Table 1 compares unfiltered LSTM errors with errors after DAE filtering on Holden.
- Short-term motion prediction: For walking, the baseline converges to a mean pose and ERD produces unnatural poses, whereas the proposed model maintains smooth, natural-looking motion over the longest reported horizon.The qualitative comparison concerns longer-horizon predictions, including the 1000 ms horizon mentioned in the text.
- A metric for motion synthesis: A pretrained activity classifier evaluates generated sequences over long horizons; the model preserves the correct class longer for cyclic walking, but eating is initially confused with similar sitting motion.The classifier-based protocol is intended to assess drift and motion degradation when ground truth is unavailable.
5. Conclusion
The paper concludes that DAE-LSTM predicts natural human motion using a 3-layer LSTM and dropout autoencoder, while introducing long-horizon evaluation. It outperforms the LSTM3LR baseline and ERD, but lacks physics-based feedback and has activity-recognition limitations.
- Model: DAE-LSTM combines a 3-layer LSTM with a dropout autoencoder trained by randomly removing joints to learn the human skeleton's spatial dependencies.The model predicts motion temporally while implicitly learning skeletal structure spatially.
- Limitations: The method's activity-recognition analysis excludes discussion and smoking because those actions are indistinguishable from other activities in the classifier's action set.This limits interpretation of that analysis to the remaining evaluated activities.
- Results: Figure 5 shows that generated eating motion eventually becomes correctly classified as eating, while the baseline converges to an ambiguous mean pose.The model's arm motions distinguish eating from sitting over time.
- Evaluation: The proposed evaluation protocol analyzes synthetic-motion quality over long horizons, where Euclidean distance to the seed is not meaningful.It is designed for generated sequences whose ground truth may be unavailable.
- Results: DAE-LSTM outperforms the LSTM3LR baseline and ERD across experiments on two datasets and multiple activities.The reported experiments include varied motion types rather than a single activity.
- Limitations: Without physics-based feedback, the model lacks concepts of mass, balance, and gravity, causing slight backward leaning and physically implausible orientation errors.The authors leave incorporation of a physical model for future work.
Supplementary
The supplementary material adds training details and experiments focused on long-term prediction, training-scheme effects, and filtering noisy LSTM outputs at each time step.
- Scope: The supplementary document reports training details and additional experiments evaluating long-term motion-sequence prediction.It complements the main submission with experiments on the proposed model.
- Experiments: It specifically examines the training scheme and the benefit of filtering noisy LSTM output at every time step.The supplementary material also refers readers to a video for qualitative comparisons over longer horizons.
6. Dropout autoencoder Training
The dropout autoencoder is trained to recover plausible human poses from corrupted inputs, using distorted joint information to learn skeletal structure implicitly.
- Training: Training uses a two-stage process that separately trains LSTM3LR and the dropout encoder before fine-tuning.Training stops once validation error converges.
- Recovery: The autoencoder recovers plausible poses from drastically distorted initial poses, even though recovered poses are not identical to the originals.This recovery helps prevent LSTM3LR from accumulating errors drastically.
7. Long-term motion prediction
Over long horizons, the proposed model continues generating natural motion where comparison models may converge to mean poses or drift into unnatural poses. It remains plausible for both walking and eating sequences.
- Long-term prediction: LSTM3LR converges to a mean pose and ERD drifts to unnatural poses, while the proposed model continues generating natural walking sequences.The supplementary video compares these behaviors from 00:10 to 00:35.
- Long-term prediction: For the aperiodic eating task, the proposed model continues generating plausible poses over the displayed long horizon.The supplementary video shows this behavior from 01:10 to 01:32.
8. Action class probabilities
Because stochastic human motion makes direct ground-truth comparisons potentially misleading, the evaluation emphasizes high-level naturalness and action-class probabilities. The classifier becomes confident once distinctive motion features appear.
- Direct comparisons with ground-truth motion can misrepresent prediction quality because human motion is stochastic.
- The evaluation therefore favors a separate action classifier over Euclidean error on ground-truth samples.
- Classifier probabilities are initially distributed similarly across classes, then concentrate on the corresponding class as distinctive features emerge.
9. Extensions
The model can follow user-provided global orientation inputs despite not being trained for this task, while external orientation control remains future work. This suggests applications in motion prediction and real-time synthesis.
- External control of pose orientation remains an identified direction for future work.
- The model follows user inputs despite not being trained specifically for externally controlled orientation.The demonstrated behavior occurs from 01:43 to 02:38 in the video.
- User-provided global orientation can drive a humanoid skeleton in any direction.
- The demonstrated capability indicates potential use in motion prediction and real-time synthesis.