Source-linked AI summary
PoseGPT: Quantization-based 3D Human Motion Generation and Forecasting
Thomas Lucas, Fabien Baradel, Philippe Weinzaepfel, Grégory Rogez
TL;DR
Human motion generation must support action-conditioned futures both with and without past observations, while continuous autoregressive predictions can become unrealistic. PoseGPT compresses motion into quantized latent sequences and predicts their next indices. The paper reports state-of-the-art performance across HumanAct12, BABEL, and GRAB.
Problem
Human motion generation must accommodate action-conditioned futures, but prior approaches either require observed past motion or generate from action labels and duration alone.
Source: §1 · p. 2Method
PoseGPT quantizes motion into causal discrete latent sequences and uses a GPT-like autoregressive model to predict next indices.
Source: §1 · p. 3, §3.1 · p. 6Results
PoseGPT achieves state-of-the-art performance on HumanAct12, BABEL, and GRAB, and outperforms existing approaches while offering more flexible conditioning.
Source: §1 · p. 3, §5 · p. 15Takeaways & Limitations
PoseGPT can generate realistic and diverse human motion from an action and duration, optionally conditioned on past observations of arbitrary length.
Source: §5 · p. 15Sample-quality metrics that compare generated samples with training data can reward overfitting; the paper uses unseen BABEL test data for most reported metrics.
Source: §4.1 · p. 10
Abstract
from arXiv · showhide
We address the problem of action-conditioned generation of human motion sequences. Existing work falls into two categories: forecast models conditioned on observed past motions, or generative models conditioned on action labels and duration only. In contrast, we generate motion conditioned on observations of arbitrary length, including none. To solve this generalized problem, we propose PoseGPT, an auto-regressive transformer-based approach which internally compresses human motion into quantized latent sequences. An auto-encoder first maps human motion to latent index sequences in a discrete space, and vice-versa. Inspired by the Generative Pretrained Transformer (GPT), we propose to train a GPT-like model for next-index prediction in that space; this allows PoseGPT to output distributions on possible futures, with or without conditioning on past motion. The discrete and compressed nature of the latent space allows the GPT-like model to focus on long-range signal, as it removes low-level redundancy in the input signal. Predicting discrete indices also alleviates the common pitfall of predicting averaged poses, a typical failure case when regressing continuous values, as the average of discrete targets is not a target itself. Our experimental results show that our proposed approach achieves state-of-the-art results on HumanAct12, a standard but small scale dataset, as well as on BABEL, a recent large scale MoCap dataset, and on GRAB, a human-object interactions dataset.
1 Introduction
Human motion generation must accommodate multiple possible futures, yet existing approaches commonly require past observations or regress continuous poses that can become unrealistic. PoseGPT instead compresses motion into causal discrete latents and predicts their indices, supporting generation with or without observed motion.
1 Introduction
- Continuous-space autoregressive models are costly at high frame rates and can produce averaged, unrealistic poses that drift over long horizons.The paper identifies these as limitations of applying autoregressive models directly to human motion.Source: §1 · p. 2
- PoseGPT uses a causal, quantized autoencoder to compress motion and preserve the temporal structure needed to condition on past observations of arbitrary length.Causal attention ensures each latent representation depends only on the present and past.Source: §1 · p. 2
- A GPT-like model predicts distributions over next discrete indices, allowing probabilistic future-motion modeling with or without past-motion conditioning.Discretization reduces low-level redundancy so the model can focus on long-range relations.Source: §1 · p. 3
- PoseGPT outperforms existing approaches while offering more flexible conditioning, according to the paper’s evaluation protocol.The protocol evaluates classifier-based sample quality, overfitting, and likelihood-based mode coverage.Source: §1 · p. 3
- The evaluation combines classifier-based sample quality, checks for overfitting, and likelihood-based measures of mode coverage.The authors report learning from substantially more data than prior work using BABEL alongside HumanAct12.Source: §1 · p. 3
2 Related work
Prior work addresses forecasting from past motion and generation from action labels, but these settings leave limitations in conditioning length, data scale, and pose representation. PoseGPT combines discrete latent modeling with autoregressive prediction to address these settings.
Human motion forecasting
- Forecasting methods can struggle with long horizons and averaged poses, while action-conditioned generation methods have faced limited datasets and restricted observation conditioning.The paper contrasts arbitrary-length past conditioning with methods that condition only per frame or lack diverse trajectories.Source: §2 · pp. 3–4
Human motion synthesis
- PoseGPT targets action-conditioned generation trained on large-scale, long-tailed data and optionally conditioned on past observations of arbitrary length.The related-work discussion identifies BABEL as a recently released dataset enabling this direction.Source: §2 · p. 4
Pose representation
- The paper learns and quantizes a pose representation rather than enforcing a specific skeleton, joint-location, or bone-angle format.Its related-work discussion notes that no unified human pose representation format has emerged.Source: §2 · p. 4
Generative modeling
- PoseGPT combines quantized autoencoder latents with an autoregressive transformer that sequentially predicts discrete latent sequences.The paper positions this as using both autoencoder and autoregressive modeling paradigms.Source: §2 · p. 4
- The paper argues that GAN mode collapse makes GANs alone unsuitable when human-robot interaction requires coverage of likely future-motion modes.The related-work discussion identifies mode coverage as critical in that application.Source: §2 · p. 5
3 The PoseGPT model
PoseGPT first learns a causal, quantized latent representation of motion, then trains an autoregressive transformer to predict latent indices. Decoding sampled index sequences produces motion conditioned on action, duration, and optionally observed motion.
3 The PoseGPT model
- PoseGPT has two stages: compress and reconstruct motion in a discrete space, then predict next latent indices with a GPT-like model.The model can condition generated motion on an action, sequence duration, and optionally an observed motion.Source: §3 · p. 5, Figure 4 · p. 7
3.1 Learning a discrete latent space representation
- The encoder maps motion into a temporally reduced latent sequence, and causal attention ensures each latent prefix depends only on its corresponding motion prefix.The decoder reconstructs motion from quantized latents; causal encoding makes past-conditioned sampling possible.Source: §3.1 · pp. 5–6, Figure 3 · p. 6
- The encoder and decoder use a straight-through gradient estimator and a commitment loss to train through nondifferentiable quantization and stabilize training.The commitment loss is identified as necessary for stable training.Source: §3.1 · pp. 6–7
- Product quantization splits each latent element into K chunks, discretizes them with K codebooks, and expands the discrete space to C^K combinations.An autoregressive head predicts the K targets sequentially to capture their relations.Source: §3.1 · p. 7
3.2 Learning a density model in the discrete latent space
- A GPT-like transformer learns the likelihood of each next latent index given prior indices, action, and sequence length.The model is trained by maximum-likelihood estimation and motion is generated by sampling indices and decoding them.Source: §3.2 · pp. 7–8
4 Experiments
Experiments evaluate PoseGPT’s latent-space and GPT design choices, sample quality, diversity, and robustness, then compare it with prior methods on HumanAct12, BABEL, and GRAB.
4 Experiments
- HumanAct12 has 1,191 videos across 12 actions, while BABEL has 28K sequences, 120 actions, and a train/validation/test split; GRAB covers object-grasping motions.HumanAct12 lacks train/validation/test splits; GRAB contains 1,500 sequences averaging eight seconds, with seven people for training and two for testing.Source: §4 · pp. 8–9
4.1 Evaluation metrics
- The evaluation pairs classifier-based quality and diversity measures with reconstruction error and GPT likelihood; BABEL metrics use unseen test data to help detect over-fitting.The RT ratio is the only reported metric insensitive to over-fitting; FID, RS, and maximum-likelihood metrics are computed using BABEL test data.Source: §4.1 · pp. 9–10
4.2 Ablative study of design choices
- On BABEL, higher latent capacity lowers PVE but does not monotonically improve RS or RT; intermediate K,C settings offer the best trade-offs.K=1 has poor test reconstruction and lower RS, while the highest-capacity models, such as K=8, perform sub-par.Source: §4.2 · p. 11
- Embedding action at each timestep improves performance, length conditioning helps, and an MLP prediction head followed by an autoregressive head yields further gains.Concatenation trains faster than summation; its additional capacity is not beneficial on HumanAct12. The prediction heads help model correlations among product-quantized indices.Source: Table 3 · p. 11, §4.2 · p. 12
- Causal attention mildly degrades performance in the encoder, where it enables conditioning on past observations; applying it in the decoder enables online operation but strongly degrades performance.The ablation studies K=2 and K=4 with C=256.Source: §4.2 · p. 12, Table 4 · p. 13
- Longer observed motion monotonically improves RS and FID, while changing softmax temperature trades sample quality against mode dropping.When repeatedly conditioning on the last generated pose, FID initially degrades but then remains stable across many generation cycles.Source: Figure 9 · p. 12, §4.2 · p. 13, Figure 10 · p. 13
4.3 Comparison to the state of the art
- Qualitative samples show diverse future motions from an initial pose and action, and different actions produce different motions from the same initial pose.The examples include two possible futures for jumping and stretching, and four motions conditioned on four actions.Source: §4.3 · pp. 13–14, Figure 11 · p. 14, Figure 12 · p. 14
- PoseGPT improves on ACTOR by 33% in FID on HumanAct12 (0.12 vs. 0.08), gains over 50% on BABEL, and substantially improves all metrics on BABEL and GRAB.On BABEL, gains are around 50% in both FID and classification accuracy; the accuracy gains indicate higher-quality samples and a richer distribution.Source: §4.3 · p. 13, Table 5 · p. 14
5 Conclusion
PoseGPT generates realistic and diverse 3D human motion from an action, duration, and optionally an arbitrarily long past observation, reaching state-of-the-art performance on three benchmarks.
5 Conclusion
- PoseGPT reaches state-of-the-art performance on three benchmarks and generates motion conditioned on an action label and arbitrarily long past observations.It also supports generation without past observations and for varying sequence lengths.Source: §5 · p. 15
A Appendix
The appendix details PoseGPT’s autoregressive prediction head, input conditioning, architecture, training, and action classifiers.
Auto-regressive prediction head
- The autoregressive prediction head models correlations among product-quantized codebook indices with little overhead compared with a naive approach.It uses K − 1 extra linear layers; the naive approach would increase network cost by K^2 and sampling time by K.Source: A Appendix · p. 20
Input embedding
- Input embeddings combine motion, position, action, and sequence length, with concatenation and a learned projection performing best among tested conditioning strategies.The alternatives include a separate action-and-length token or adding conditioning at every time step by summing embeddings.Source: A Appendix · p. 20
Architectural details
- The encoder, decoder, and autoregressive model are transformers, with the encoder using three blocks of four transformer layers and the autoregressive model using eight layers.The decoder mirrors the encoder’s architecture in reversed block order; the autoregressive model uses four attention heads per layer.Source: A Appendix · pp. 20–21
Training details and action classifiers
- Both training stages run for 2 million iterations, and the full procedure takes 3 days on one Nvidia V100 GPU.For efficiency, vertex reconstruction losses are applied to a random 10% to 20% of frames; HumanAct12 uses the classifier from, while BABEL and GRAB use classifiers trained by the authors.Source: A Appendix · p. 21