Source-linked AI summary
Human Motion Prediction via Spatio-Temporal Inpainting
Alejandro Hernandez Ruiz, Juergen Gall, Francesc Moreno-Noguer
TL;DR
The paper targets long-term 3D human-motion forecasting that preserves both body motion and absolute position while addressing shortcomings of L2 evaluation. It uses spatio-temporal inpainting with a multi-discriminator GAN and reports realistic, distributionally faithful predictions that remain robust to corrupted observations.
Problem
Long-term motion predictors often omit absolute body position and use L2, which inadequately captures realistic motion distributions.
Method
STMI-GAN represents camera-frame 3D skeleton sequences as masked spatio-temporal tensors and uses a time-preserving convolutional generator with three discriminators.
Results
The approach produces long-term predictions beyond 2 seconds, including full-body pose, orientation, and position, and is robust to missing frames and occluded joints.
Takeaways & Limitations
Frequency-distribution metrics and adversarial training evaluate and encourage multiple realistic futures rather than exact framewise matches.
Abstract
from arXiv · showhide
We propose a Generative Adversarial Network (GAN) to forecast 3D human motion given a sequence of past 3D skeleton poses. While recent GANs have shown promising results, they can only forecast plausible motion over relatively short periods of time (few hundred milliseconds) and typically ignore the absolute position of the skeleton w.r.t. the camera. Our scheme provides long term predictions (two seconds or more) for both the body pose and its absolute position. Our approach builds upon three main contributions. First, we represent the data using a spatio-temporal tensor of 3D skeleton coordinates which allows formulating the prediction problem as an inpainting one, for which GANs work particularly well. Secondly, we design an architecture to learn the joint distribution of body poses and global motion, capable to hypothesize large chunks of the input 3D tensor with missing data. And finally, we argue that the L2 metric, considered so far by most approaches, fails to capture the actual distribution of long-term human motion. We propose two alternative metrics, based on the distribution of frequencies, that are able to capture more realistic motion patterns. Extensive experiments demonstrate our approach to significantly improve the state of the art, while also handling situations in which past observations are corrupted by occlusions, noise and missing frames.
1. Introduction
The paper addresses limitations in long-term 3D human-motion forecasting by jointly predicting body pose and absolute position through spatio-temporal inpainting, while evaluating motion distributions beyond L2 similarity.
- Motivation: Existing methods simplify forecasting by omitting global body position, requiring action labels, and relying heavily on L2 distance.L2 can favor static mean-pose predictions and is inaccurate for comparing long motion sequences.
- Approach: The proposed GAN represents camera-frame skeleton observations as a spatio-temporal tensor and formulates prediction as inpainting missing volumes.Its fully convolutional generator preserves temporal coherence while three discriminators enforce realistic skeleton and motion properties.
- Approach: The model forecasts non-rigid body pose and absolute position for long horizons of 2 seconds or more without ground-truth annotations for generated frames or explicit action information.Future frames are controlled by discriminators, while L2 is applied only to reconstructed past observations.
- Evaluation: The paper introduces a metric that compares distributions over the human motion manifold rather than seeking exact jointwise agreement across every frame.This targets the distributional structure of realistic motion instead of the pointwise objective used by L2.
- Results: Experiments report full-body pose, orientation, and position forecasting that remains robust to missing frames and occluded joints in past observations.Figure 1 illustrates a generated motion that differs from the ground truth while remaining semantically indistinguishable.
2. Related Work
Prior work uses recurrent, convolutional, and encoder-decoder models for motion prediction, but commonly evaluates long-term motion with L2 and omits global translation and rotation. The paper connects motion forecasting with structured inpainting, drawing on GAN-based image completion while addressing temporal prediction.
- Deep Learning for motion prediction: Deep motion-prediction methods commonly use RNNs, sequence-to-sequence models, encoder-decoder predictors, or autoregressive CNN generators.Representative approaches include ERD-LSTM, structural RNNs, and models combining L2, adversarial, or geodesic losses.
- Deep Learning for motion prediction: These methods often evaluate motion with L2, which fails to capture motion semantics for long-term predictions, and joint-angle parameterizations do not estimate body rotation and translation in space.The related-work discussion identifies both metric and representation limitations.
- Sequence completion and image inpainting: Sequence-completion methods based on low-rank factorization or RNNs are not designed for future prediction.This distinguishes generic sequence completion from the forecasting problem addressed by the paper.
- Sequence completion and image inpainting: Denoising and variational autoencoders cannot handle large missing portions of structured data, whereas conditional GANs have advanced image inpainting.The paper draws inspiration from GANs conditioned on partial or corrupted images.
- Metrics for evaluating human motion prediction: Alternative motion metrics include learned contrastive metrics and distribution-based measures inspired by image-generation evaluation.The learned metric is semantically strong but difficult to apply across models after training; image-generation work motivates distributional evaluation.
- Connection to the proposed approach: The proposed architecture overview combines a masked 3D-joint sequence, a fully convolutional time-preserving generator, geometric losses, and three independent discriminators.This architecture connects the inpainting formulation with adversarial motion evaluation.
3. Problem Formulation
Human motion is represented as a tensor of camera-frame 3D joint coordinates, and a binary mask specifies which coordinates are unobserved. Estimating the masked coordinates unifies forecasting and several occlusion or missing-frame settings.
- Representation: A motion sequence of F skeletons with J joints is represented as S ∈ R^F×J×3, using 3D Cartesian coordinates in the camera reference frame.Rotation and translation are inherently encoded in this coordinate representation.
- Masking: A binary mask M ∈ B^F×J×3 identifies unobserved sequence parts, producing the masked input through the element-wise product S ◦ M ≡ S_m.The model estimates the 3D coordinates corresponding to masked joints.
- Sub-problems: Masking the last frames expresses forecasting, while masking intermediate joints represents random or structured occlusions and missing frames.Different mask patterns define related prediction sub-problems within the same formulation.
- Sub-problems: The model can handle any combination of these forecasting and occlusion sub-problems.
4. Model
STMI-GAN formulates 3D human motion prediction as spatio-temporal inpainting, using a convolutional generator and multiple discriminators to reconstruct visible observations and generate realistic future motion.
- Architecture: STMI-GAN represents motion prediction as inpainting missing portions of a spatio-temporal skeleton tensor.The input is a masked sequence, and the network hypothesizes the missing volume.
- Architecture: The frame encoder maps each pose into a time-invariant embedding before generator CNNs process temporal and embedded spatial dimensions.The encoded sequence retains temporal ordering and represents occluded frames with constant zero values.
- Architecture: Three discriminator branches assess the generated sequence, skeleton geometry through an Euclidean Distance Matrix, and motion characteristics.The EDM representation is rotation- and translation-invariant, focusing discrimination on skeleton shape.
- Losses: Reconstruction loss preserves visible observations while avoiding penalties on alternative future completions.The reconstruction objective is applied only where the original sequence is visible.
- Losses: Limb-distance and bone-length losses encourage accurate semantic joint spacing and well-formed skeletons across generated frames.The limb loss uses relative distances, while the bone-length loss enforces constant lengths over the generated sequence.
- Losses: The full objective combines reconstruction, limb, bone, discriminator, and generator losses through tunable hyperparameters.The generator and discriminator are trained in a minimax formulation.
5. Metrics for motion prediction
The paper evaluates motion distributions in the frequency domain rather than relying only on pointwise L2 distance, using entropy and KL-divergence measures over power spectra.
- Frequency-based metrics: PSEnt measures the entropy of the power spectrum computed independently for each joint and coordinate axis.The metric provides a rough estimate of model fitness from the frequency distribution of generated motion.
- Frequency-based metrics: Low power-spectrum entropy indicates a biased model, whereas high entropy can indicate noisy, inaccurate, or unstable motion.L2-trained generators commonly regress toward the mean and thereby reduce generated-sequence entropy.
- Frequency-based metrics: PSKL measures KL-divergence between the power-spectrum distributions of ground-truth and generated datasets.Because KL-divergence is asymmetric, both PSKL(GT, Gen) and PSKL(Gen, GT) are computed.
- L2-based metrics: The reported L2 baseline measures joint-wise distances between ground-truth and generated sequences, with the dataset mean used as the aggregate measure.In this work, sequences are represented in Cartesian coordinates readable in millimeters.
6. Implementation Details
Implementation uses frame-wise fully connected encoding, U-block generation with skip connections, residual discriminator CNNs, and spatial alignment for absolute-coordinate motion.
- Frame Autoencoder: The frame encoder is a fully connected network with sequential blocks containing two layers and an attention mechanism.The attention mask is produced from each block's input through a linear transformation and sigmoid activation.
- Generator U-Blocks: Generator U-blocks downsample with convolutions, upsample with transposed convolutions, and use skip connections to preserve sequence shape.The representation is compressed to a small spatial resolution before being restored to the input dimensions.
- Residual CNN: Each of the three discriminator branches uses a separate residual CNN classifier with convolutional blocks and additive residual connections.Intermediate block outputs are transformed, concatenated, and passed to the final classifier output.
- Spatial Alignment: Spatial alignment subtracts the first-frame hip position and rotates the skeleton to a common facing direction before encoding.The alignment layer is reversed after frame decoding so predictions return to the absolute coordinate system.
7. Experiments
Experiments evaluate motion prediction against baselines using power-spectrum metrics, L2 comparisons, human judgments, qualitative examples, and occlusion completion. STMI-GAN preserves motion-distribution statistics over time, produces diverse predictions despite higher L2, and remains robust under severe occlusion.
- Motion prediction: 1.65 PSEnt over four seconds indicates long-term motion has a biased, more complex frequency distribution rather than uniform short-term frequencies.PSEnt is approximately 0.678 in one-second windows but rises to 1.65 over four seconds.
- Motion prediction: STMI-GAN combines three discriminator types and closely approximates the expected motion distribution without performance decay over time.Its PSEnt remains close to ground truth and PSKL stays low, indicating similar motion amount and kind.
- Motion prediction: L2 Res.sup. →(0.69, 0.36, 0.64, 0.25) versus L2 STMI-GAN →(1.09, 0.74, 1.33, 0.96), yet STMI-GAN predictions appear more diverse and realistic.The experiments attribute this mismatch to different objectives: spatial-distance minimization versus reproducing the human-motion distribution.
- Motion prediction: Human evaluations found baseline and STMI-GAN average scores in a similar range, with STMI-GAN slightly better, while trained evaluators judged baselines poorly.Scores above 50% indicate generated motion was judged more realistic than ground truth; the highest absolute-prediction score occurred for STMI-GAN.
- Motion prediction: The qualitative examples show successful action and direction guesses, but prediction speed can remain inaccurate when motion becomes uncertain.A stopped person before forecasting creates many plausible futures, increasing uncertainty.
- Occlusion completion: With 80% occlusion, the generator is particularly robust to structured occlusions, while random occlusions can also be handled by linear interpolation depending on their nature.The completion task conditions on only 20% of the data.
8. Conclusions
The paper presents a GAN for long-term 3D human motion prediction, including absolute body position, using spatio-temporal inpainting and frequency-distribution guidance.
- The GAN forecasts 3D human motion beyond 2 seconds, including the body's absolute position.
- The method formulates prediction as inpainting in spatio-temporal volumes of historical skeleton poses.
- Three independent discriminators guide generated sequences toward the dataset's human-motion frequency distribution.
- New frequency-based metrics address L2's inadequacy by representing multiple possible futures through dataset frequency distributions.
- Experiments on Human3.6M show the model generates highly realistic human motion predictions.