PaperlayerView exact revision on arXiv

Source-linked AI summary

TEMOS: Generating diverse human motions from textual descriptions

Mathis Petrovich, Michael J. Black, Gül Varol

arXiv:2204.14109v2cs.CVcs.CL

TL;DR

Text-conditioned human-motion generation must produce plausible 3D pose sequences while handling ambiguity in language, but prior methods commonly generate only one motion per description. TEMOS uses a cross-modal variational model with Transformer encoders to sample diverse motions, including SMPL bodies, and reports state-of-the-art results on KIT. Its main scope boundaries are limited vocabulary and locomotive coverage, unprincipled diversity measurement, typo sensitivity, and quadratic-memory limits for very long motions.

  • Problem

    Text-conditioned motion generation requires plausible 3D human motions from language, while many prior methods produce only one output despite descriptions admitting multiple valid performances.

  • Method

    TEMOS uses a cross-modal VAE with Transformer encoders for text and motion, sampling latent vectors to generate diverse motions through a text-to-motion branch.

  • Results

    TEMOS achieves state-of-the-art results on KIT, substantially outperforming prior methods on nearly all reported metrics and in perceptual studies.

  • Takeaways & Limitations

    The framework supports diverse text-conditioned motion generation across skeleton-based and expressive SMPL body representations.

  • Takeaways & Limitations

    The model is limited by KIT’s small locomotive-heavy vocabulary, lacks a principled diversity metric, is sensitive to typos, and cannot scale to very long motions because of quadratic memory cost.

Abstract

from arXiv · show

We address the problem of generating diverse 3D human motions from textual descriptions. This challenging task requires joint modeling of both modalities: understanding and extracting useful human-centric information from the text, and then generating plausible and realistic sequences of human poses. In contrast to most previous work which focuses on generating a single, deterministic, motion from a textual description, we design a variational approach that can produce multiple diverse human motions. We propose TEMOS, a text-conditioned generative model leveraging variational autoencoder (VAE) training with human motion data, in combination with a text encoder that produces distribution parameters compatible with the VAE latent space. We show the TEMOS framework can produce both skeleton-based animations as in prior work, as well more expressive SMPL body motions. We evaluate our approach on the KIT Motion-Language benchmark and, despite being relatively straightforward, demonstrate significant improvements over the state of the art. Code and models are available on our webpage.

1 Introduction

TEMOS addresses text-conditioned 3D human-motion generation by modeling language ambiguity and producing diverse motions, including expressive SMPL sequences. It uses a variational, Transformer-based framework and reports strong benchmark performance despite evaluation challenges.

  • Motivation: Text-conditioned motion generation supports virtual and real-world applications, but motion capture is expensive and motivates automatic synthesis.Applications include film, games, and speech-controlled robotic assistance.
  • Approach: TEMOS encodes language and motion with Transformers in a joint latent space and uses a variational formulation for generation.The approach is trained with motion and text branches, reconstruction losses, and latent-variable sampling.
  • Motivation: Most prior methods produce one motion per text, although ambiguous descriptions can correspond to multiple plausible performances.TEMOS instead samples a distribution of motions conditioned on natural-language descriptions.
  • Capabilities: TEMOS generates SMPL body-model sequences in addition to conventional skeleton-based animations.SMPL provides body surfaces that may support motion involving objects or scenes.
  • Evaluation: The framework reports large state-of-the-art gains, while evaluation remains difficult because quantitative metrics often assume one true motion per description.The authors therefore combine ground-truth comparisons with human perceptual studies.

2 Related work

Related work spans unconstrained and conditioned motion synthesis, including text-to-motion systems built from sequence-to-sequence or joint cross-modal representations. TEMOS distinguishes itself through variational diversity, Transformer sequence encoding, and unified body-motion handling.

  • Human motion synthesis: Motion synthesis includes unconstrained generation and conditioned synthesis using signals such as music, speech, action, or text.Unconstrained models can sample diverse motions but lack controllability.
  • Text-conditioned motion generation: Text-conditioned methods include sequence-to-sequence formulations and joint cross-modal embeddings that map text and motion into a shared space.These approaches build on advances in natural-language modeling and cross-modal representation learning.
  • Motion representation: Some prior systems use impoverished representations by omitting global trajectory or modeling only upper-body motion.Such choices can make motions unrealistic or ignore global movement described by text.
  • Limitations and distinction: Many state-of-the-art text-conditioned models are deterministic, limiting their ability to generate multiple motions from one description.TEMOS addresses this limitation with variational sampling in a shared cross-modal space.
  • TEMOS: TEMOS combines Transformer sequence-level motion encoding with VAE distribution parameters and avoids separately hand-crafting upper- and lower-body representations.Its framework also supports text-conditioned SMPL body-surface synthesis.

3 Generating multiple motions from a textual description

TEMOS generates multiple plausible 3D human-motion sequences from text by mapping text and motion into a shared variational latent space. A Transformer decoder reconstructs motions non-autoregressively, while cross-modal training aligns the two modalities and supports variable-duration generation.

  • Motion representations: The model supports both skeleton-based and SMPL body-motion representations for generating 3D human motions.Quantitative evaluation uses known ground-truth durations, while inference can specify the desired duration.
  • Model design: TEMOS represents text and motion in a joint latent space using Transformer-based encoders within a variational autoencoder formulation.The text encoder uses pretrained DistilBERT word embeddings, while the motion encoder processes pose sequences.
  • Model design: A Transformer decoder generates a complete motion sequence non-autoregressively from one latent vector and a specified duration.This design promotes globally consistent motion and permits variable durations as an additional diversity source.
  • Training strategy: During training, text-motion pairs pass through text-to-motion and motion-to-motion branches, with both branches reconstructing the 3D poses.At test time, only the text-to-motion branch is used.
  • Training strategy: The loss combines reconstruction, KL regularization, and cross-modal embedding similarity to align text and motion distributions in the latent space.The text and motion embeddings are regularized toward a standard normal distribution and encouraged to be close to each other.

4 Experiments

Experiments on KIT compare TEMOS with prior methods, perceptual judgments, ablations, and SMPL generations. TEMOS improves benchmark performance and human-rated quality, while its evaluation and data scope impose limitations.

  • Data and evaluation metrics: Evaluation is suboptimal because KIT usually provides one ground-truth motion per text, limiting distribution-based metrics for a model intended to generate multiple motions.The paper therefore combines single-sample and best-of-multiple quantitative evaluation with perceptual studies.
  • Comparison to the state of the art: TEMOS substantially outperforms prior text-conditioned motion-generation methods on most KIT metrics, including when sampling one random motion per text.The comparison uses Lin et al., JL2P, and Ghosh et al.; the exception is APE on local joints.
  • Comparison to the state of the art: Users preferred TEMOS over Lin et al. for matching descriptions 90.5% of the time, and preferred TEMOS over ground truth 15.5% of the time against Ghosh et al.’s 8.5%.The perceptual study separately evaluated text-motion correspondence and realism.
  • Comparison to the state of the art: TEMOS was preferred for realism over ground truth 38.5% of the time, compared with 5.5% for Ghosh et al.The realism study hid the textual description from participants.
  • Ablation study: Generating 10 motions per text lets the best candidate improve metrics over single-generation settings, while random and mean-latent single samples perform similarly.The deterministic variant removes sampling and KL losses; the variational model supports random or zero-vector generation and best-of-10 evaluation.
  • Ablation study: The Transformer architecture is the most critical architectural choice, whereas additional losses provide relatively minor improvements and language-model finetuning does not improve results.The language-model experiment concerns end-to-end finetuning of DistilBERT.
  • SMPL results: TEMOS produces diverse SMPL body-model generations for each text, extending evaluation beyond skeleton-based motion representations.The qualitative diversity evaluation displays two random samples per description.
  • Limitations: The KIT data has a small, locomotive-motion-dominated vocabulary, and TEMOS may fail on typos or very long motions because of quadratic memory cost.The authors report 1263 unique words and identify out-of-distribution descriptions and multi-minute motions as boundary cases.

5 Conclusion

TEMOS generates diverse 3D human motions from natural-language descriptions using a variational approach, achieving strong benchmark and perceptual results. The work also extends generation to full-body meshes and identifies future directions and privacy concerns.

  • TEMOS generates diverse 3D human motions from natural-language descriptions through a variational approach.
  • The method achieves state-of-the-art results on the KIT Motion-Language benchmark, outperforming prior work in quantitative and perceptual studies.
  • The reported improvements are mainly attributed to incorporating sequence modeling through Transformers.
  • TEMOS uses full-body meshes rather than only skeleton representations.
  • Future directions include explicit contact modeling, physics integration, and duration estimation, while applications animating people without consent may raise privacy concerns.

A Additional experiments

The additional experiments examine hyperparameter sensitivity, sampling behavior, motion representation, and training-data choices. They also document validation-based hyperparameter selection and a single-sample evaluation protocol.

  • The final hyperparameters were selected using the validation set, although they were not always optimal on the test set.
  • Evaluation uses a single random sample, while selecting among multiple generations can improve results by finding a motion closer to the ground truth.
  • The experiments also report quantitative results for a model trained with SMPL rotations.
  • Training samples longer than 500 frames after 12.5 Hz subsampling are discarded, representing about 2.3% of the training data.

A.2 Weight of the KL losses and the embedding loss

The appendix varies the KL-loss and embedding-loss weights and examines how multiple generations affect root-position error. Embedding-loss weighting has the stronger reported influence, while additional sampling reduces APE root error.

  • Embedding-loss weighting influences results more than KL-loss weighting, while performances remain similar when values are not too low.
  • A λE value of 10^-3 deteriorates performance, leading the authors to fix both λKL and λE at 10^-5.
  • Sampling more generated sequences per text reduces the APE root metric error when the best sample is selected against the ground truth.

A.3 Number of Transformer layers

The appendix studies Transformer depth, attention heads, model size, and language-model substitutions. Results are relatively insensitive to Transformer layer and head counts, while smaller configurations and pretrained-language-model choices show specific boundaries.

  • Transformer configuration: Performance is optimal when all Transformers use 6 layers and 6 heads, which is adopted in the remaining experiments.
  • Transformer configuration: Adding only 1 or 2 layers above DistilBERT is insufficient, whereas configurations beginning with 4 layers are satisfactory; the model uses 6.
  • Related ablations: The reported batch-size ablation finds the best performance at either batch size 8 or 32, with larger sizes constrained by GPU memory.
  • Transformer configuration: Results are slightly better for larger Transformer models but are not very sensitive to changes in layer and head counts.
  • Language model: DistilBERT performs similarly to BERT and RoBERTa while using fewer parameters, suggesting it is sufficient for this task.

A.5 Sampling multiple motions

TEMOS can generate multiple motions for one text description, allowing evaluation of both best-case matching and diversity rather than only a single output.

  • 1 to 10 generations per text produce a monotonic decrease in APE root error when the closest sample is selected.This reflects the benefit of choosing the generated motion closest to the ground truth.
  • APE is 0.78 in the best case, 0.96 for a random generation, and 1.24 for the most different of 10 generated motions.The worst-case value is qualified because one ground-truth motion does not represent every valid motion matching the text.
  • Adding more text-encoder layers improves results, although a lightweight model on top of the language model still performs reasonably.
  • Larger language models than DistilBERT do not produce significant performance changes.

A.6 Quantitative results with the SMPL model

The SMPL-based evaluation aligns SMPL-H joints with MMM through joint selection, canonical body shapes, and optional rescaling. The appendix also details the standardized skeleton and SMPL motion representations used by the model.

  • Evaluation setup: SMPL-H joints are mapped to an MMM-compatible skeleton subset for quantitative comparison.The correspondence is provided in Table A.6.
  • Evaluation setup: Rescaling SMPL joints by 0.64 makes performance comparable to MMM-based training on the KITSMPL test set.The model is evaluated both with and without rescaling; results are reported in Table A.7.
  • Evaluation setup: Both SMPL-H and MMM bodies use canonical standard shapes, specifically an average neutral body for SMPL-H and a robot-style body for MMM.The comparison evaluates how the rescaling affects alignment between the two representations.
  • Representation preprocessing: Both motion representations are standardized by subtracting training-set means and dividing by training-set standard deviations.The skeleton and SMPL representations are described separately but receive the same training-data normalization.
  • Skeleton representation: The skeleton representation concatenates local joint coordinates, angular changes, and root translation features into 64-dimensional frame vectors.A sequence of duration F is represented in R^F × 64, with the first orientation angle assumed to be zero during integration.
  • SMPL representation: The SMPL representation concatenates 21 local rotations, global rotation, and root translation into 135-dimensional frame vectors.Global orientation is canonicalized so that all sequences begin oriented in the same direction.

C Evaluation details

The evaluation computes motion errors after canonicalizing body orientation and reports position and variation metrics across root, trajectory, local, and global components. The appendix also documents perceptual studies and implementation issues affecting reproduced evaluations.

  • Metric preprocessing: All evaluation motions are canonicalized so the body faces the same forward direction before metrics are computed.Except for the root joint, other joints are expressed in the body’s local coordinate system.
  • Metrics: Average Position Error measures mean L2 distance between generated and ground-truth joint positions across frames and test samples.The metric is defined separately for each joint and then reported over selected joint groups.
  • Metrics: Average Variance Error measures the L2 difference between generated and ground-truth joint variances, capturing differences in motion variation.The variance is computed for each joint, alongside the joint mean used in the definition.
  • Reported breakdowns: Reported errors cover root-joint coordinates, horizontal root trajectories, mean local errors, and mean global errors.Trajectory errors use only the root joint’s X and Y coordinates.
  • Evaluation implementation: The evaluation code could not be reused because it failed to reproduce prior results and contained an author-confirmed trajectory-loss slicing bug.The issue was identified in src/eval_APE.py at line 249.
  • Perceptual studies: Perceptual studies use pairwise AMT comparisons for semantic correspondence and realism, with an additional single-video semantic rating study.The pairwise semantic setup may favor more realistic motion, motivating the additional one-video evaluation.

D Implementation details

TEMOS uses fixed Transformer settings across its encoders and decoder, and its implementation relies on common deep-learning configuration libraries. The appendix also reports training runtime and describes action-phrase grouping in the KIT Motion-Language dataset.

  • Architecture: TEMOS uses 256-dimensional embeddings, six Transformer layers, six attention heads, 0.1 dropout, and 1024-dimensional feedforward layers.These settings apply to all encoders and the decoder.
  • Runtime: Training takes about 4.5 hours for 1K epochs on a single 16GB Tesla V100, using approximately 15GB of GPU memory.The reported batch size is 32, with an average of 16 seconds per epoch.
  • Dataset processing: The KIT Motion-Language dataset contains 3911 motions and 6352 text sequences, while action phrases are extracted from verbs and their complements.This procedure groups text sequences corresponding to the same action phrase.
Loading 2204.14109v2…