Source-linked AI summary
Action-Conditioned 3D Human Motion Synthesis with Transformer VAE
Mathis Petrovich, Michael J. Black, Gül Varol
TL;DR
The paper addresses action-conditioned generation of realistic and diverse human motion without requiring an initial pose or sequence. It introduces ACTOR, a Transformer-based conditional VAE with a sequence-level latent representation that decodes variable-length SMPL motions from an action and duration. The authors report state-of-the-art action-conditioned motion-generation performance and demonstrate action-recognition augmentation and motion denoising, while noting resource-dependent maximum duration and predefined action categories.
Problem
Realistic, diverse, controllable motion synthesis remains difficult, especially when applications require a semantic action and duration without an initial motion sequence.
Method
ACTOR is a Transformer-based conditional VAE that samples a sequence-level motion latent vector and decodes SMPL motion conditioned on an action label and duration.
Results
The authors report state-of-the-art performance on action-conditioned motion generation across multiple datasets and demonstrate generated-data augmentation for action recognition plus motion denoising.
Takeaways & Limitations
The learned sequence-level latent space supports diverse action-conditioned synthesis and can serve motion denoising and action-recognition applications.
Takeaways & Limitations
Maximum generated duration depends on computational resources, and the model supports only a predefined set of actions.
Abstract
from arXiv · showhide
We tackle the problem of action-conditioned generation of realistic and diverse human motion sequences. In contrast to methods that complete, or extend, motion sequences, this task does not require an initial pose or sequence. Here we learn an action-aware latent representation for human motions by training a generative variational autoencoder (VAE). By sampling from this latent space and querying a certain duration through a series of positional encodings, we synthesize variable-length motion sequences conditioned on a categorical action. Specifically, we design a Transformer-based architecture, ACTOR, for encoding and decoding a sequence of parametric SMPL human body models estimated from action recognition datasets. We evaluate our approach on the NTU RGB+D, HumanAct12 and UESTC datasets and show improvements over the state of the art. Furthermore, we present two use cases: improving action recognition through adding our synthesized data to training, and motion denoising. Code and models are available on our project page.
1. Introduction
ACTOR addresses unconstrained action-conditioned motion synthesis by generating diverse, realistic 3D sequences from an action label and duration, without requiring an initial motion sequence. Its Transformer VAE uses a sequence-level latent representation and supports noisy monocular motion data plus action-recognition and denoising applications.
- Problem: ACTOR generates realistic, controllable 3D human motions from a semantic action label and specified duration, rather than extending an existing sequence.The target includes varying-length motions such as throwing, for applications including virtual reality and character control.
- Approach: The model uses a Transformer encoder-decoder trained with a VAE objective and SMPL body representations from action-labeled 3D motion data.SMPL supports joint- and surface-based reconstruction losses and interaction modeling through body surfaces.
- Novelty: Positional encodings let the decoder output an entire variable-length sequence at once, avoiding autoregressive drift and mean-pose regression.The method also introduces an action-conditioned sequence-level embedding instead of a frame-level latent representation.
- Data and scope: Limited labeled MoCap data motivates using monocular motion estimates, which are noisy but sufficient for training and can be denoised through encoding and decoding.The approach is demonstrated on 40 fine-grained UESTC action categories without requiring multi-view cameras.
- Results and uses: The paper reports state-of-the-art performance across multiple datasets and demonstrates use cases in action recognition augmentation and MoCap denoising.The contributions also include an architecture and loss ablation study.
2. Related Work
Prior work spans future-motion prediction, unconstrained synthesis, denoising, and Transformer VAEs, while ACTOR focuses on action-conditioned generation from sequence-level latent representations. Its design contrasts especially with frame-level or autoregressive alternatives.
- Motion prediction: Future-motion prediction typically conditions on past motion or an initial pose, whereas ACTOR targets synthesis from action categories without that initial-sequence constraint.Related approaches include statistical models, GANs, VAEs, and Transformer-based prediction systems.
- Motion synthesis: Many unconstrained synthesis methods overrepresent walking and running, while ACTOR samples broader predefined, acyclic action categories compatible with recognition datasets.This enables controlled generation for categories used in action recognition.
- Closest comparison: Action2Motion uses a GRU-based per-frame VAE, whereas ACTOR combines a Transformer architecture with a sequence-level VAE latent space.The sequence-level representation is presented as a central distinction from frame-level latent modeling.
- Applications: Related work also learns motion manifolds for denoising and uses generated motions for action-recognition augmentation, connections shared by ACTOR.Other literature includes normalizing flows and graphics methods for animator control.
- Transformer VAEs: Transformer-VAE literature has used timeframe embeddings, hidden-state averaging, or attention averaging, whereas ACTOR uses learnable tokens to pool inputs into one sequence-level embedding.The tokens are inspired by pooling mechanisms in language and vision Transformers.
3. Action-Conditioned Motion Generation
ACTOR is a conditional Transformer VAE that maps arbitrary-length SMPL pose sequences to an action-aware sequence-level latent space and decodes sampled motions for a requested duration. Training combines pose, vertex, and KL losses to balance reconstruction and latent regularization.
- Motion representation: SMPL represents pose and body location while enabling differentiable recovery of body-mesh vertices and joint coordinates from generated poses.The representation separates pose from identity-specific shape for actors with different morphology.
- Encoder: ACTOR conditions both encoder and decoder on an action category while encoding arbitrary-length pose sequences into a Gaussian latent distribution.Learnable action-specific tokens pool temporal information into distribution parameters μ and Σ, from which z is sampled.
- Decoder: The decoder generates a complete motion in one shot from a latent vector, action label, and duration, using duration-dependent sinusoidal positional encodings.The resulting T decoder outputs are projected to poses and passed through SMPL for vertices and joints.
- Reconstruction losses: Training includes an L2 pose reconstruction loss covering SMPL rotations and root translations, plus a vertex-coordinate reconstruction loss.The vertex loss compares root-centered meshes produced by a differentiable, parameter-free SMPL layer using mean shape.
- Latent regularization: The KL term regularizes the encoder distribution toward a standard Gaussian, and the total objective is L = LP + LV + λKLLKL.The weight λKL is tuned to trade off diversity and realism.
4. Experiments
The experiments evaluate datasets, reconstruction losses, architecture choices, sequence-duration handling, state-of-the-art performance, and action-recognition use cases. Results support Transformer-based design choices, variable-length generation, and applications to denoising and data augmentation.
- Ablation study: Constraining both SMPL rotations and vertex coordinates is critical for smooth motion, while coordinate-based losses alone fail to converge meaningfully on UESTC.The evaluated losses include rotation, joint-coordinate, vertex-coordinate, and combined rotation-plus-vertex terms.
- Datasets and evaluation metrics: Experiments use three temporally trimmed action-recognition datasets and report results using repeated trials with 95% confidence intervals.The datasets are NTU RGB+D, HumanAct12, and UESTC; UESTC recognition uses pose parameters expressed as 6D rotations.
- Ablation study: The Transformer model outperforms fully connected and GRU encoder-decoder architectures on two datasets, while learnable distribution and action tokens materially affect performance.Replacing the distribution parameter tokens causes a considerable drop; replacing the additive action token with one-hot conditioning especially harms UESTC performance.
- Ablation study: Fixed-size training retains over 85% accuracy across [40, 120] frames, while variable-duration training improves performance and reduces degradation beyond the seen range.Variable-duration training uses sequences sampled between 60 and 100 frames and is initialized through 60-frame pretraining before finetuning.
- Comparison to the state of the art: Compared with Action2Motion and other baselines, ACTOR achieves significant improvements on NTU-13 and HumanAct12 while directly producing human meshes.Action2Motion instead fits SMPL models to generated joint coordinates through an optimization step.
- Use cases in action recognition: Encoding-decoding can denoise motion estimates, and generated motions provide useful augmentation for action recognition, especially in low-data regimes.The experiments also report 97.0% versus 91.8% for one denoising comparison, while generated-only training reaches 80.7% on the real test set.
- Qualitative results: Generated motions preserve action semantics while varying hands, angles, speed, phase, and action-irrelevant body parts.Qualitative examples include left- and right-handed executions of the ‘Throw’ action.
- Qualitative results: The maximum generated duration depends on computational resources, and the action vocabulary remains restricted to a predefined set.The authors identify open-vocabulary actions as future work.
5. Conclusions
The paper presents a Transformer-based VAE for action-conditioned human-motion synthesis and reports state-of-the-art performance. It also explores motion denoising and action-recognition use cases, with a sequence-level latent space enabling future applications.
- ACTOR is a Transformer-based VAE for synthesizing action-conditioned human motions.
- The method achieves state-of-the-art performance on action-conditioned motion generation, significantly improving over prior work.
- The authors evaluate applications in motion denoising and action recognition.
- Its sequence-level latent space is identified as a basis for future priors in motion estimation or action recognition.
A. Additional experiments
The additional experiments ablate the model and vary key parameters to assess how design choices influence result quality.
- The experiments vary λKL, batch size, number of layers, and SMPL pose-parameter rotation representation.
A.1. Weight of the KL loss
The KL-loss weighting controls a trade-off between motion diversity and realism, with λKL = 1e−5 selected for all experiments.
- λKL = 1e−5 best balances diversity and realism and is used in all experiments.
A.2. Influence of the batch size
Batch size significantly influences performance, and the authors select 20 based on the reported comparison.
- Batch size 20 gives the best performance among the tested sizes and is used in all experiments.The tested batch sizes are 10, 20, 30, and 40 with a fixed learning rate.
A.3. Number of layers
The Transformer VAE performs best with eight layers in both encoder and decoder, although gains beyond six layers are marginal.
- Eight Transformer layers are used in both the encoder and decoder because performance degrades at two or four layers and improves only marginally beyond six.The difference between six and eight layers is minimal.
A.4. SMPL pose parameter representation
The study compares SMPL rotation representations and selects the 6D continuous representation for UESTC, where it performs best; axis-angle is difficult to train.
- Axis-angle rotation representations are difficult to train because of discontinuities.
- The 6D continuous representation performs best on UESTC and is used in all experiments.On NTU-13, quaternions, rotation matrices, and 6D representations have similar performance.
B. Additional qualitative results
Additional qualitative experiments show diverse action-conditioned generations and examine reconstruction losses, durations, latent interpolation, and denoising behavior.
- ACTOR generates diverse motions for additional actions on NTU-13 and UESTC.The experiments show three generations per action and qualitatively illustrate different ways to perform the same action.
- The supplementary experiments compare combined rotation-and-vertex reconstruction losses with single-loss alternatives and vary generation duration.
- Latent-space interpolation and encoding-decoding experiments inspect controllability and denoising, with the latter producing smoother, natural-looking motion from jerky inputs.
- Removing temporal jitter from Action2Motion worsens its evaluation results, changing FID from 0.41 to 0.63 and accuracy from 94.3% to 93.0%.The evaluation uses the authors’ own script and normalized ground-truth SMPL shape.
C. Implementation details
The implementation uses a fixed Transformer VAE configuration, trains with selected loss and batch-size settings, and evaluates SMPL representations according to dataset-specific action-recognition inputs.
- The Transformer uses 256-dimensional embeddings, eight layers, four attention heads, 0.1 dropout, a 1024-dimensional feedforward network, and GELU activations.
- Models are pretrained on fixed-duration sequences and then fine-tuned for variable durations for 100 additional epochs with the same hyperparameters.The UESTC training setup uses 1K epochs on a single Tesla V100 GPU with 4GB memory.
- Training uses KL-loss weight λKL = 1e−5 and batch size 20 to balance diversity and realism and accommodate training sensitivity.
- The six- or eight-layer Transformer configurations are preferred because performance degrades at two or four layers and gains after six are marginal.
- The 6D continuous SMPL representation is best on UESTC, while NTU-13 shows similar performance for most non-axis-angle representations.Generations are converted to the representations used by each dataset’s action-recognition evaluator.
- Additional qualitative results cover four action categories from NTU-13 and UESTC, with three generations shown per action.