Source-linked AI summary
MT-VAE: Learning Motion Transformations to Generate Multimodal Human Dynamics
Xinchen Yan, Akash Rastogi, Ruben Villegas, Kalyan Sunkavalli, Eli Shechtman, Sunil Hadap, Ersin Yumer, Honglak Lee
TL;DR
Human motion generation must capture structured, multimodal transitions between short-term motion modes. MT-VAE learns motion embeddings and transformations between modes, then samples transformations to generate future motions. Across facial and full-body motion, it produces diverse and plausible sequences and supports motion transfer and video synthesis.
Problem
Human motion is multimodal: one initial motion mode may transition into several plausible future modes, which deterministic models may collapse into a single mode.
Method
MT-VAE jointly learns reconstructable feature embeddings for motion modes and feature transformations representing transitions between them, with stochastic transformation sampling.
Results
MT-VAE generates diverse and plausible future facial and full-body motions and supports analogy-based motion transfer and future video synthesis.
Takeaways & Limitations
A structured motion-mode representation with stochastic transformations can model human dynamics across facial and full-body motion using minimal supervision.
Takeaways & Limitations
The approach assumes motion modes have low-dimensional representations and transitions can be modeled as feature transformations; a larger sequence-level cycle-consistency loss was ineffective in preliminary experiments.
Abstract
from arXiv · showhide
Long-term human motion can be represented as a series of motion modes---motion sequences that capture short-term temporal dynamics---with transitions between them. We leverage this structure and present a novel Motion Transformation Variational Auto-Encoders (MT-VAE) for learning motion sequence generation. Our model jointly learns a feature embedding for motion modes (that the motion sequence can be reconstructed from) and a feature transformation that represents the transition of one motion mode to the next motion mode. Our model is able to generate multiple diverse and plausible motion sequences in the future from the same input. We apply our approach to both facial and full body motion, and demonstrate applications like analogy-based motion transfer and video synthesis.
1 Introduction
MT-VAE models long-term human motion as motion modes connected by transformations, enabling stochastic generation of diverse future facial and full-body motions.
- Problem: Human actions can be segmented into motion modes that capture short-term dynamics and stylistic attributes, with transitions connecting modes over time.Examples include transitions from neutral expression to smiling to laughing, or distinct walking styles.
- Method: MT-VAE jointly learns sequence-level motion embeddings and feature transformations representing transitions between present and future motion modes.Its LSTM encoder-decoder embeds short subsequences into reconstructable feature vectors, while transformations connect current and future modes.
- Contributions: The model learns motion representations and transformations without supervision, using a structured sequence of motion modes and transitions.The contribution includes a generative motion model trained in an unsupervised manner.
- Multimodal generation: Stochastic sampling of the transformation space generates multiple plausible future motion modes from one input, addressing multimodal transitions that deterministic models may collapse.The approach can represent transitions that are rare or absent in the training set.
- Applications and evaluation: Experiments on facial and full-body motion report results that are more plausible and diverse than previous work, while supporting analogy-based motion transfer and future video synthesis.The paper defines plausibility as preserving human-dynamics structure and diversity as exploring multiple motion modes.
2 Related Work
Prior work spans motion prediction, video generation, feature transformations, and multimodal conditional generation, while highlighting persistent diversity and mode-collapse challenges.
- Motion prediction: Earlier human-motion research emphasized hierarchical spatio-temporal representations, while later work used deep networks to predict and synthesize future motion.Related approaches include predicting dense future flow from a single image.
- Video generation: Video-generation methods include patch-based short-term synthesis and action-conditioned architectures that provide semantic control through conditions such as rotation and location.These methods target future video generation from one or more initialization frames.
- Feature transformations: Feature transformations have been studied in word embeddings, paired-image reconstruction or prediction, and transformation-aware video representations.These works motivate representing relations between examples through embedding transformations.
- Multimodal generation: Multimodal conditional generation has been explored across images, sketches, language, and video, but diverse plausible generation remains difficult because mode and posterior collapse can occur.Collecting sequence data with one-to-many pairs is also described as non-trivial.
3 Problem Formulation and Methods
The paper formulates future motion prediction as mapping a current motion mode to a future mode, then introduces MT-VAE to represent motion modes and their stochastic transformations in latent space. Its variants add structured transformations, cycle consistency, and motion coherence to improve the representation and generation process.
- 3 Problem Formulation and Methods: Given a motion mode S_A of T observations, the task is to predict a future mode S_B whose first step follows the current sequence.Each observation is a D-dimensional vector, and the modes encode changes in the moving object.
- 3.1 Preliminaries: The deterministic encoder-decoder LSTM maps the input sequence to a fixed-length feature and decodes that feature into a future sequence.The decoder is initialized from the encoded feature and additionally receives x_T for smoother prediction.
- 3.1 Preliminaries: A vanilla conditional VAE introduces a sampled latent variable z so one input motion mode can generate multiple possible future sequences.Its decoder conditions on both the input feature e_A and z, but the semantic role of z may not represent long-term motion trends clearly.
- 3.2 Motion-to-Motion Transformations in Latent Space: MT-VAE explicitly models each motion mode as a low-dimensional feature and represents transitions between modes as transformations of those features.The model contains an LSTM encoder, a latent transformation encoder, a latent decoder producing e*_B, and an LSTM sequence decoder.
- 3.2 Motion-to-Motion Transformations in Latent Space: In MT-VAE, the latent transformation is inferred from current and future motion features, decoded together with e_A, and used to synthesize the future feature and sequence.The sequence decoder receives the synthesized feature and the same additional current observation used by the prediction baseline.
- 3.3 Additive Transformations in Latent Space: MT-VAE (add) constrains transformations by encoding T = e_B − e_A, reconstructing T* from z and e_A, and applying an additive interaction to obtain e*_B.This variant replaces concatenated feature encoding with an explicit difference representation intended to capture the mode transition.
- 3.3 Additive Transformations in Latent Space: Cycle consistency regularizes the feature transformation by requiring a transformed latent code to reconstruct the original sampled transformation.The paper defines separate cycle losses for the concatenated and additive MT-VAE variants.
- 3.3 Additive Transformations in Latent Space: A motion coherence loss encourages smooth velocity during the first K prediction steps, while the full objective combines VAE, cycle-consistency, and motion-coherence losses.The authors report that a larger sequence-level cycle was ineffective in their recurrent setting and hypothesize vanishing or exploding gradients as a reason.
4 Experiments
Experiments evaluate MT-VAE on facial and full-body motion using quantitative metrics, qualitative comparisons, human judgments, ablations, and motion-transfer applications. Across these evaluations, MT-VAE generates diverse, plausible motions with natural transitions and supports analogy-based transfer and future video synthesis.
- Experimental Setup: Experiments use Aff-Wild for facial motion and Human3.6M for full-body motion, with LSTM-based sequence encoder and decoder components.Aff-Wild contains facial expression-pose data, while Human3.6M provides 2D keypoint trajectories; the model uses 1-layer LSTMs with 1,024 hidden units.
- Quantitative Evaluation: Evaluation measures reconstruction and sampling quality using R-MSE, S-MSE, and conditional log-likelihood, with multiple samples generated for stochastic models.R-MSE measures the closest reconstruction to ground truth, while S-MSE evaluates the closest generated sample; evaluations use 50 samples for R-MSE and 500 for S-MSE and log-likelihood.
- Quantitative Evaluation: The MT-VAE (add) model achieves the best quantitative performance among latent-variable models, with lower reconstruction error and similar sampling error than MT-VAE (concat).The result suggests that the additive formulation further regularizes motion-transformation learning.
- Qualitative Results: Qualitative comparisons show that MT-VAE generates multiple diverse and plausible futures with more natural transitions than the compared baselines.Vanilla VAE sometimes misses the motion trend and produces artifacts, whereas MT-VAE better preserves motion structure across facial and full-body examples.
- Human Evaluation: Human evaluations find that MT-VAE provides a balance between realism and diversity, unlike baselines that favor one criterion.Prediction LSTM is judged most realistic but least diverse, while the stochastic video-prediction baseline is most diverse but least realistic.
- Ablation and Applications: Ablations show that removing cycle consistency or motion coherence reduces reconstruction performance, while context-free transformation modeling produces poor S-MSE.The results support the role of these components in preserving discriminative motion structure and modeling transformations conditioned on the input feature.
- Ablation and Applications: MT-VAE transfers a motion-mode transition from sequences A to B onto sequence C and also supports diverse future pixel-level video generation.In analogy-based transfer, MT-VAE combines the learned transition with sequence C's structure, while the video application generates diverse and realistic future frames.
5 Conclusions
The paper learns conditional human-motion generation by structuring long-term dynamics as motion modes linked by transitions. This structure supports stochastic generation across facial and full-body motion representations.
- MT-VAE models long-term human dynamics as motion modes with transitions between them.The architecture regularizes this structured space for conditional motion generation.
- The model can learn conditional human-motion generation with minimal supervision by enforcing strong structure.
- Stochastic sampling in the structured motion space supports generation of multiple possible motion futures.
- The same approach models both facial and full-body motion across representations such as shape parameters and keypoints.