Source-linked AI summary
Deep representation learning for human motion prediction and classification
Judith Bütepage, Michael Black, Danica Kragic, Hedvig Kjellström
TL;DR
Existing 3D human-motion models often cover few activities and may not generalize to new movements. The paper trains bottleneck feed-forward temporal encoders on broad mocap data to predict future poses and learn transferable features, reporting strong motion prediction and classification utility.
Problem
Human-motion models are often restricted to small activity sets, while skeletal data requires representations that capture temporal structure and limb correlations for novel movements.
Method
Bottleneck feed-forward temporal encoders predict future mocap frames from recent history, using symmetric, time-scale, or body-hierarchy network structures.
Results
The learned representation supports action classification and motion prediction, with feed-forward encoders reported to outperform recurrent approaches and state-of-the-art prediction methods.
Takeaways & Limitations
Generic deep feed-forward representations trained on mocap can serve as both feature extractors and generative models for human motion.
Takeaways & Limitations
Prediction error increases over longer horizons, and the relative effects of network structure and parameter count require further investigation.
Abstract
from arXiv · showhide
Generative models of 3D human motion are often restricted to a small number of activities and can therefore not generalize well to novel movements or applications. In this work we propose a deep learning framework for human motion capture data that learns a generic representation from a large corpus of motion capture data and generalizes well to new, unseen, motions. Using an encoding-decoding network that learns to predict future 3D poses from the most recent past, we extract a feature representation of human motion. Most work on deep learning for sequence prediction focuses on video and speech. Since skeletal data has a different structure, we present and evaluate different network architectures that make different assumptions about time dependencies and limb correlations. To quantify the learned features, we use the output of different layers for action classification and visualize the receptive fields of the network units. Our method outperforms the recent state of the art in skeletal motion prediction even though these use action specific training data. Our results show that deep feedforward networks, trained from a generic mocap database, can successfully be used for feature extraction from human motion data and that this representation can be used as a foundation for classification and prediction.
1. Introduction
The paper seeks a general, efficient human-motion representation that transfers across tasks and novel movements despite skeletal data’s distinctive temporal and limb structure. It proposes bottleneck feed-forward temporal encoders trained for future-frame prediction and evaluates their learned features for classification and prediction.
- Motivation: Human-motion representations should support classification, prediction, generation, and transfer to novel movements and applications.The representation must encode both joint/limb correlations and temporal structure without being tuned to specific motion patterns.
- Motivation: Image-style convolutions are unsuitable for mocap because meaningful correlations follow limbs and time rather than local spatial neighborhoods.Filters therefore need to span the joints and convolve only temporally.
- Motivation: Recurrent models can struggle with aperiodic complex movements, contributing to action-specific training rather than broad generalization.Walking is periodic, but many complex human movements are not.
- Approach: The proposed bottleneck networks predict future mocap frames from recent history while investigating time-scale and body-hierarchy structural priors.The study compares learned representations through visualization, action classification, and motion prediction.
- Contributions: The scheme is trained on a large CMU mocap corpus to learn a generic representation, with reported uses in classification and motion prediction.The authors describe it as unsupervised representation learning for long-term prediction not confined to a small action set.
2. Related work
Prior skeletal-motion methods commonly specialize in recognition or restrict prediction to limited motions, while recurrent and convolutional designs impose practical constraints. This work instead targets generic motion representation and reports broader prediction capability and state-of-the-art performance.
- Action recognition: Supervised HMM, multilayer-perceptron, and hierarchical-RNN methods focus on action recognition, whereas this work learns representations for prediction and uses recognition mainly for validation.These prior methods model action-dependent or hierarchical temporal dynamics directly for classification.
- Representation learning: The comparable DSAE reconstructs 0.2 seconds from three recordings containing seven motion sequences, while this work targets general motion across many everyday actions.The paper reports that its method outperforms the DSAE in Section 4.2.
- Motion prediction: Earlier prediction models were restricted to walking, jogging, and running, motivating a model that captures a larger variety of actions.The paper contrasts its goal with autoregressive Restricted Boltzmann Machine experiments limited to those motions.
- Architectural limitations: Convolutional pooling can blur features and limit deeper hierarchies, while recurrent encoder-decoder and structural-RNN approaches may become task-tuned or computationally complex.The cited recurrent approaches model temporal dynamics but can require separate models or high-complexity structural graphs.
- Positioning: The proposed simple motion-dynamics representation is reported to outperform state-of-the-art motion-prediction methods and enable a wider range of motions.The authors connect this broader scope with applications such as human-robot interaction.
3. Methodology
The method encodes a window of past skeletal observations into a low-dimensional bottleneck and decodes it into future observations. Three temporal-encoder variants impose symmetric, time-scale, or body-hierarchy structure while accommodating mocap’s temporal and skeletal organization.
- Data processing: The skeleton is represented in Cartesian coordinates after centering joint positions and subtracting each trial’s mean pose while preserving global rotation.Joint angles are converted using a standardized body model.
- Data representation: Each mocap training example concatenates ∆t frames into an input window followed by an output window of the same size.The frame representation has dimension 3 × Njoints, so each window has dimension 3 × Njoints × ∆t.
- Temporal encoding: A temporal encoder maps the last ∆t observations into a low-dimensional representation and decodes that representation into the next ∆t observations.The encoder and decoder may use different structures because encoding local features and decoding a globally valid structure impose different requirements.
- Network structure: S-TE mirrors the encoder in the decoder, C-TE combines temporal convolutions at different scales, and H-TE incorporates the human-body hierarchy.The figure depicts fully connected layers in blue and time-only convolutional layers in green.
- Network structure: H-TE builds a tree from joints to limbs and ultimately the whole body, using selective parent-layer connections trained jointly with the temporal encoder.The hierarchy encodes structural relationships between body parts.
4. Experiments
Experiments show that the learned temporal representations encode interpretable motion structure, support action classification, and predict both specific and general motions over long horizons. Hierarchical structure and joint-window prediction improve robustness, including for missing limbs.
- Feature visualization: Middle-layer units encode whole-body rotations, postures, and individual limb movements, indicating that the representation captures distinct motion patterns.Spike-triggered averages were computed from poses eliciting sigmoid-unit outputs above 0.8.
- Feature visualization: GPFA trajectories separate walking from boxing and waving, while representing turning degree and the two-loop structure of figure-eight motion.Repetitive boxing and waving patterns are reflected in latent space and are separated from walking because they primarily involve arm movements.
- Action classification: Temporal encodings achieved classification rates comparable to or slightly higher than deep sparse autoencoders despite reducing dimensionality from 7200 inputs to 100-dimensional middle layers.The authors interpret this as evidence that temporal encoding extracts action-relevant information in a compact representation.
- Motion prediction: 160 ms and longer predictions favored temporal encoders over LSTM3L, which performed better for some initial predictions.Jointly predicting an entire time window reduces diffusion and propagated errors; H-TE generally outperformed S-TE and C-TE, while action-specific fine-tuning helped especially for long-term prediction and unseen actions such as smoking.
- Motion prediction: On a walking sequence, H-TE-F closely matched ground truth across 1600 ms, whereas the general model remained accurate for approximately 600 ms before diverging while staying meaningful to human observers.Predictions beyond 560 ms can diverge substantially because human motion is stochastic.
- Generalization and missing data: General temporal encoders stayed close to single-action performance while recurrent models generalized less; C-TE favored short-term information and S-TE was more suited to long-term prediction.For missing limbs during eating, prediction error increased by about one decimal but remained comparatively low, allowing the models to infer the missing pose without significant divergence.
5. Discussion
The temporal encoder framework learns a structured, lower-dimensional representation that supports both prediction and classification. Feed-forward encoders outperform recurrent approaches in reported prediction settings, but model differences and increasing long-term error remain unresolved limitations.
- The learned representation occupies a structured, lower-dimensional space spanning diverse motion patterns and supports action classification without fine-tuning.The representation is intended to function both as a generative model and a feature extractor.
- Feed-forward temporal encoders outperform recurrent approaches for short- and long-term prediction and generalize to novel subjects and actions.
- The bottleneck structure and simple feed-forward design may drive performance by forcing an efficient and sufficient data representation.
- Feed-forward networks require a pre-specified input window, although sliding-window long-term predictions are reported as more accurate than recurrent approaches.
- The relative performance of symmetric, convolutional, and hierarchical models may reflect differences in parameter count and network structure, while prediction error increases over longer horizons.The authors identify uncertainty modeling and systematic investigation of window size as future work.