Source-linked AI summary
TEACH: Temporal Action Composition for 3D Humans
Nikos Athanasiou, Mathis Petrovich, Michael J. Black, Gül Varol
TL;DR
The paper addresses the problem of generating 3D human motions for temporally ordered sequences of natural-language actions, a setting not handled well by existing single-action methods. TEACH uses BABEL and a hierarchical Transformer approach that generates each action non-autoregressively while proceeding autoregressively across actions. It improves quantitatively over independent and joint baselines, supports variable numbers of actions, and produces fewer transition discontinuities, although acceleration peaks and ambiguous overlapping BABEL segments remain limitations.
Problem
Temporal action composition requires generating 3D motions for ordered natural-language action sequences, whereas prior text-conditioned methods primarily handle single actions and face data and scalability constraints.
Method
TEACH recursively generates one motion per action, conditioning the current text and past frames through a variational encoder-decoder while retaining non-autoregressive generation within each action.
Results
TEACH improves quantitatively over independent and jointly trained baselines, while supporting variable numbers of actions and producing fewer transition discontinuities.
Takeaways & Limitations
The work establishes temporal action composition as a benchmarked text-to-motion task using BABEL’s diverse sequential action annotations.
Takeaways & Limitations
TEACH remains susceptible to acceleration peaks at action transitions, and overlapping BABEL segments can make action sequences visually unclear.
Abstract
from arXiv · showhide
Given a series of natural language descriptions, our task is to generate 3D human motions that correspond semantically to the text, and follow the temporal order of the instructions. In particular, our goal is to enable the synthesis of a series of actions, which we refer to as temporal action composition. The current state of the art in text-conditioned motion synthesis only takes a single action or a single sentence as input. This is partially due to lack of suitable training data containing action sequences, but also due to the computational complexity of their non-autoregressive model formulation, which does not scale well to long sequences. In this work, we address both issues. First, we exploit the recent BABEL motion-text collection, which has a wide range of labeled actions, many of which occur in a sequence with transitions between them. Next, we design a Transformer-based approach that operates non-autoregressively within an action, but autoregressively within the sequence of actions. This hierarchical formulation proves effective in our experiments when compared with multiple baselines. Our approach, called TEACH for "TEmporal Action Compositions for Human motions", produces realistic human motions for a wide variety of actions and temporal compositions from language descriptions. To encourage work on this new task, we make our code available for research purposes at our $\href{teach.is.tue.mpg.de}{\text{website}}$.
1. Introduction
TEACH introduces temporal action composition: generating 3D human motion from ordered natural-language prompts and durations. It combines BABEL’s diverse motion-text data with recursive, past-conditioned generation to support variable-length sequences and smoother transitions.
- Task: Temporal action composition generates 3D human motion for a sequence of natural-language descriptions in temporal order.The task focuses on sequential rather than simultaneous action composition.
- Data: BABEL provides larger, more diverse free-form language annotations than prior motion-text datasets, enabling broader action coverage.The work directly trains on English descriptions rather than only categorical action labels.
- Approach: TEACH recursively generates one action at a time while conditioning each action on the previous motion, avoiding the long-sequence scaling issues of non-autoregressive generation.Within each action, generation remains non-autoregressive and can produce diverse motions.
- Approach: Encoding the previous action’s last few frames and applying short-window Slerp address discontinuities between generated actions.The authors report that this transition strategy treats transitions better than Slerp applied between independently generated actions.
- Contributions: TEACH establishes a benchmark and is designed to generate variable numbers of actions with smooth transitions from a large action vocabulary.The paper reports promising text-to-motion results for this setting.
2. Related Work
Prior motion-synthesis research spans prediction, infilling, semantic conditioning, and environmental constraints, but much of it emphasizes locomotion or single-action generation. TEACH extends language-conditioned synthesis to sequential action compositions using BABEL and conditioning on previous actions.
- Prior motion synthesis: Existing motion-synthesis work includes prediction, infilling, action-label or language conditioning, and pose or environment constraints.Recent methods include both autoregressive variable-length models and methods that generate complete sequences at once.
- Semantic scope: Much prior infilling and synthesis work focuses on locomotion, while human behavior includes broader actions and complex combinations.This motivates attention to richer semantic control.
- Language-conditioned generation: Language-conditioned methods map descriptions to poses or motions using RNNs, joint embedding spaces, separated body encoders, and other architectures.These approaches generally address semantic control from language rather than temporal action sequences.
- Closest prior work: TEMOS uses a Transformer-VAE for free-form text-conditioned motion generation, whereas TEACH uses the larger and more diverse BABEL dataset for sequential actions.The paper identifies TEMOS as the closest related method but notes that it cannot synthesize action sequences directly.
- TEACH: TEACH explicitly conditions each generated action on the previous one to model actions performed sequentially.This property is tied to the sequential action annotations available in BABEL.
3. Motion Synthesis with TEACH
TEACH generates action sequences by conditioning each action on its text description and recent preceding motion, using paired action data from BABEL for training.
- 3.2. Architecture: TEACH extends text-conditioned motion synthesis to sequences by using a Past Encoder that conditions each new action on the previous action’s final frames.The first action uses text and learnable tokens, while subsequent actions additionally encode past motion features.
- 3.2. Architecture: The architecture encodes current instructions with frozen DistilBERT and combines text, past-motion features, and learnable tokens in a Transformer encoder.The encoder outputs Gaussian parameters µi and Σi, which define the distribution used to sample and decode each motion.
- 3.3. Training: Training uses BABEL action segments and extracts overlapping action pairs, including triples connected through transitions, for models beyond the Independent baseline.The Independent baseline uses single training segments, whereas the remaining models train on extracted action pairs.
- 3.3. Training: Each training iteration generates two motions sequentially, conditioning the second on the second instruction, its length, and the last P frames of the first generated motion.The two forward passes are optimized jointly using reconstruction and KL losses.
- 3.3. Training: The total objective combines reconstruction and KL losses, regularizing the two latent Gaussian distributions toward a standard normal distribution.The loss is L = LR + λKLLKL, with λKL = 10^-5 in practice.
4. Experiments
Experiments on BABEL evaluate dataset coverage, sequence-generation variants, transition handling, past-motion conditioning, qualitative behavior, and limitations. TEACH is compared with independent and joint baselines and supports diverse action compositions with smoother, more coherent transitions.
- Dataset: BABEL provides 10,881 motion sequences and 65,926 textual labels, with overlapping annotated segments that support sequence-of-actions generation.The experiments use processed textual descriptions and exclude categorical action labels.
- Baselines: The independent baseline generates each action separately, while joint training combines descriptions and requires paired consecutive-action motions.Independent generation applies Slerp after generation, whereas joint training models two-action pairs directly.
- Comparison with baselines: TEACH recursively conditions each future action on previous motion, using the last five frames, and outperforms independent and joint baselines on BABEL validation metrics.The evaluation samples one variational generation and reports APE and AVE on root and body joints.
- Qualitative analysis and limitations: Qualitative results show TEACH completing sequential hand actions naturally, covering diverse non-locomotive actions, and generating sequences longer than two actions.The paper also reports acceleration peaks at transitions and ambiguity from overlapping BABEL action segments.
- Transition interpolation: Slerp reduces transition discontinuities, while aligning the second motion is crucial for the independent baseline; TEACH produces smoother transitions than the baseline.Without alignment, the reported transition distance comparison is 0.177 to 0.122 in favor of TEACH.
5. Conclusions
The paper introduces temporal action composition on BABEL and reports that TEACH improves over baselines while supporting variable action counts and fewer transition discontinuities. It also identifies realism, world contact, and future-aware generation as areas for improvement.
- TEACH improves quantitatively over independent and joint baselines while allowing variable numbers of actions and producing fewer transition discontinuities.
- The task and benchmark target motion generation from sequences of textual prompts, extending beyond single-action text-conditioned synthesis.
- Motion realism could be improved, and contact with the world could be modeled explicitly.
- The current setting assumes the character only looks backward in time rather than using knowledge of future actions.
C. Dataset statistics
The supplementary material points to additional statistics for BABEL, whose annotated data underpin the paper’s temporal action composition experiments.
- BABEL statistics are provided for different label types and dataset properties in the supplementary material.
C.1. Language statistics
BABEL contains substantially richer language than KIT, including more frequent occurrences of KIT’s common tokens across action-relevant part-of-speech combinations.
- BABEL is at least twice as rich in language as KIT according to token-frequency analysis.
- For KIT’s most frequent tokens and action-relevant part-of-speech combinations, the same tokens occur much more often in BABEL.
C.2. Duration statistics
BABEL distinguishes full-sequence labels from shorter extracted segments and consecutive segment pairs used for temporal action composition.
- Sequences are AMASS motions with one action label, Segments are shorter motions extracted from longer sequences, and Pairs combine consecutive segments.
- Consecutive Segments are used to build action pairs, while Sequences are included in independent model training.
- Segment durations have a mean three times smaller than Sequence durations, with medians consistently below means, indicating long-tailed duration distributions.
D.1. Data processing
The data-processing setup uses BABEL’s original training and validation splits, with validation results reported because the test set is unavailable. Motions are subsampled at 30 fps, filtered by duration, canonicalized, and standardized according to the training configuration.
- BABEL’s original training and validation splits are used, with final results reported on validation because the test set is not publicly available.
- AMASS motions are subsampled at 30 fps before training.
- Motion pairs shorter than 0.3 seconds or longer than 25 seconds are removed, while motion segments or sequences shorter than 0.3 seconds are also filtered.
- Input bodies are rotated to face forward, with each action canonicalized separately for the independent baseline and entire sequences canonicalized from the first frame for TEACH and the joint baseline.
- The data are standardized separately for paired-motion and single-action training cases.
D.2. Alignment & interpolation
The alignment baseline connects independently generated actions by matching their boundary poses and interpolating between them. This step is crucial because independent generation does not ensure continuity across transition poses.
- The second motion is translated and rotated so its first frame matches the first action’s last frame before interpolation.
- Slerp interpolates between the aligned poses, inserting 8 frames between consecutive actions.
- Alignment is crucial for the independent baseline because its separately generated actions do not ensure continuity within transition poses.