Source-linked AI summary
Synthesis of Compositional Animations from Textual Descriptions
Anindita Ghosh, Noshaba Cheema, Cennet Oguz, Christian Theobalt, Philipp Slusallek
TL;DR
The paper addresses how to generate plausible 3D human motions from natural-language descriptions, including complex sentences with sequential and superimposed actions. It proposes a hierarchical two-stream model with separate upper- and lower-body motion representations and contextualized language embeddings. The method significantly outperforms prior state-of-the-art methods on quantitative metrics and qualitative evaluations, while generalization to novel superimpositions remains imperfect.
Problem
Text-to-motion systems struggle to map long, complex sentences describing multiple sequential or simultaneous actions into correct 3D pose sequences.
Method
A hierarchical two-stream sequential model learns separate upper- and lower-body motion manifolds and maps contextualized language representations to pose sequences.
Results
The method significantly outperforms Ahuja et al. and Lin et al. on quantitative metrics and qualitative evaluations.
Takeaways & Limitations
The framework synthesizes plausible motions for short single-action sentences and longer compositional sentences with sequential or superimposed actions.
Takeaways & Limitations
Generalization to novel superimposed actions is not always successful when those combinations were not encountered during training.
Abstract
from arXiv · showhide
"How can we animate 3D-characters from a movie script or move robots by simply telling them what we would like them to do?" "How unstructured and complex can we make a sentence and still generate plausible movements from it?" These are questions that need to be answered in the long-run, as the field is still in its infancy. Inspired by these problems, we present a new technique for generating compositional actions, which handles complex input sentences. Our output is a 3D pose sequence depicting the actions in the input sentence. We propose a hierarchical two-stream sequential model to explore a finer joint-level mapping between natural language sentences and 3D pose sequences corresponding to the given motion. We learn two manifold representations of the motion -- one each for the upper body and the lower body movements. Our model can generate plausible pose sequences for short sentences describing single actions as well as long compositional sentences describing multiple sequential and superimposed actions. We evaluate our proposed model on the publicly available KIT Motion-Language Dataset containing 3D pose data with human-annotated sentences. Experimental results show that our model advances the state-of-the-art on text-based motion synthesis in objective evaluations by a margin of 50%. Qualitative evaluations based on a user study indicate that our synthesized motions are perceived to be the closest to the ground-truth motion captures for both short and compositional sentences.
1. Introduction
The paper targets text-to-motion synthesis for complex sentences describing sequential or simultaneous actions, using a hierarchical two-stream architecture to generate plausible 3D pose sequences. It combines joint-level motion representations, contextualized language embeddings, and adversarial training, and reports significant gains over prior methods.
- Motivation: Complex textual descriptions can specify multiple sequential or simultaneous actions, directions, frequencies, and other circumstances that must be mapped into pose sequences in the correct order.Existing methods may handle only one action or produce incorrect compositional motions, especially beyond locomotion.
- Proposed Contributions: The hierarchical joint embedding space separates motion into upper-body and lower-body embeddings, then further organizes them into limb embeddings.Sentence semantics such as speed, direction, and frequency are mapped to temporal pose sequences through combined embeddings.
- Proposed Contributions: The sequential two-stream autoencoder combines body-part-specific layers into upper- and lower-body manifold representations.The authors report reduced smoothing of upper-body movements and more robust synthesized motion.
- Proposed Contributions: BERT embeddings with handpicked word features are used to improve understanding of contextualized textual descriptions.The approach differs from prior methods that do not use a contextualized language model.
- Proposed Contributions: Additional loss terms condition velocity and motion-manifold learning, while a pose discriminator with adversarial loss improves motion plausibility.These components are added during network training.
- Evaluation: The method significantly outperforms the state-of-the-art methods of Ahuja et al. and Lin et al. on quantitative metrics and qualitative evaluations.The evaluations compare both objective benchmarks and user-perceived motion quality.
2. Related Work
Prior work models human motion with data-driven temporal, graph, and recurrent architectures, and conditions pose prediction on poses, audio, speech, images, or language. Text-to-action systems include statistical mappings, shared text-animation latent spaces, and instruction-based virtual-agent models.
- Data-driven Motion Modeling: Data-driven human-motion synthesis has used temporal convolutional, graph convolutional, and recurrent neural networks.Pose forecasting predicts short- or long-term future pose sequences from motion history or latent representations.
- Conditioned Motion Forecasting: Other approaches forecast poses from pixels, locomotion trajectories, audio, speech, or images.Examples include audio-conditioned dance generation and speech-conditioned upper-body behaviors.
- Text-Based Motion Synthesis: Language-to-action research has mapped linguistic instructions to virtual-agent actions using deep learning, Hidden Markov Models, bigram statistics, and shared text-animation latent spaces.These approaches include generating animations from text and actions from natural-language descriptions of video data.
3. Proposed Method
The proposed method maps language and motion into a joint, hierarchically organized latent space with separate upper- and lower-body streams, then decodes language-conditioned pose sequences recurrently. Training combines reconstruction, embedding-similarity, velocity, and adversarial objectives to improve correspondence and plausibility.
- Joint embedding: The model jointly embeds natural-language sentences and pose sequences using separate upper-body and lower-body representations.The pose encoder maps motion into two manifold vectors, while the sentence encoder produces corresponding two-stream sentence embeddings.
- Pose decoder: The recurrent hierarchical decoder generates pose sequences from pose or language embeddings, recursively using each generated pose to produce the next time step.Language-conditioned output is denoted P̂^s, and the final sentence-based sequence is P̂ = P̂^s.
- Pose encoder: The hierarchical pose encoder decomposes the skeleton into five major parts and combines their features into upper- and lower-body manifold representations.The parts are the left arm, right arm, trunk, left leg, and right leg; recurrent units produce the final representations.
- Sentence encoder: The sentence encoder uses selected BERT hidden layers and a two-layer LSTM to represent syntax, semantics, and long-range dependencies in complex sentences.Layers associated with subjects, adverbs, verbs, and prepositional objects are concatenated before LSTM encoding.
- Optimization: Training combines smooth ℓ1 pose and manifold reconstruction losses with velocity, embedding-similarity, and adversarial losses.A discriminator distinguishes real from generated poses, while the weighted objective jointly trains the pose autoencoder, sentence encoder, and discriminator.
4. Experiments
The experiments evaluate the model on KIT Motion-Language using quantitative metrics, ablations, and a user study. The evaluation measures pose accuracy, distributional variation, embedding alignment, and style similarity.
- Dataset: KIT Motion-Language contains 3,911 whole-body motion recordings and 6,278 natural-language annotations, split into training, validation, and test sets at 0.6:0.2:0.2.The dataset includes simple actions, complicated movements, and longer descriptions of multiple actions.
- Quantitative Evaluation Metrics: APE measures average joint-position differences, while AVE measures differences between generated and ground-truth joint variances.The evaluation separates local joint-position errors from global root-trajectory errors because text descriptions permit different plausible motions.
- Quantitative Evaluation Metrics: CEE measures the difference between pose and sentence manifold representations to assess how well the joint embedding space correlates corresponding modalities.The pose representations come from the pose encoder, while sentence representations come from the sentence encoder.
- Quantitative Evaluation Metrics: SEE compares Gram matrices of sentence and pose embeddings to account for general style information in motion and language.The metric computes the distance between corresponding sentence-embedding and pose-embedding Gram matrices.
- Ablation and User Evaluation: The ablation evaluation uses 23 participants ranking 14 motion videos from the proposed method, four ablations, and ground-truth captures for textual correspondence, quality, and naturalness.One ablation removes joint training of the pose-language embedding space by training model components in stages.
5. Results and Discussion
Compared with JL2P and Lin et al., the proposed method improves objective motion-synthesis metrics by more than 50% and produces qualitatively closer motions for complex actions. User-study evidence is reported against ablations, while comparisons with the state-of-the-art methods were excluded from that study.
- Objective Evaluation: 55.4% lower mean local-joint APE than JL2P and 58.4% lower than Lin et al. were achieved.Including the global trajectory, mean APE improvements were 55.7% over JL2P and 58.7% over Lin et al.
- User Study: The state-of-the-art methods were excluded from the user study after participants reported that the proposed method beat them in obvious ways.The user study instead compared the proposed method with four model ablations.
- Evaluation Caveat: Reported APE numbers for the state-of-the-art methods differed from the original paper because the authors could not replicate those numbers using the provided code and models.The comparison used pre-trained models and test-set results.
- Objective Evaluation: CEE and SEE improved by 50% over JL2P, and by 72.3% and 83.1%, respectively, over Lin et al.These results support stronger correspondence between generated poses and their input sentences.
- Qualitative Results: The method generated more accurate kicks, waltz steps, trajectories, walking directions, and turning motions than the benchmark methods in qualitative examples.Benchmark outputs omitted actions, arm movements, correct walking directions, circular trajectories, or turning motions in the reported examples.
6. Limitations, Future Work and Conclusion
The framework advances text-based motion synthesis, including compositional actions, but generalization to novel superimposed actions remains limited. Future work targets zero- or few-shot synthesis, longer multi-person narration, and improved physical plausibility.
- Conclusion: The model advances state-of-the-art text-based motion synthesis on qualitative evaluations and several objective benchmarks.
- Limitations: Generalization to novel superimposed actions is not always successful because the model accurately synthesizes superimpositions encountered during training.
- Future Work: The authors propose zero- or few-shot learning to generate simultaneous actions without training on those specific combinations.
- Future Work: Future directions include narration-based transcripts for long, step-by-step actions involving multiple people and physical constraints to improve foot sliding, limb constraints, and biomechanical plausibility.
- Conclusion: Figure 6 compares generated animations for long sentences specifying direction and step count using trajectories, start and end points, and footstep markers.
- Conclusion: Modeling a variety of motions and complex sentence structures is presented as an essential step toward realistic mixed-action animations and broader practical applicability.
Appendix: More Results on Quantitative Evaluation Metrics
The appendix reports positional, variance, content, and style metrics for the model, baselines, and ablations. Although some ablations achieve lower overall positional error, root-joint and variance measures favor the final model's motion quality.
- APE: 1.96% lower mean APE with global trajectory is achieved by the ‘w/o 2-St’ ablation, but averaging reduces relevant joint movements.
- APE: The final model has the lowest root-joint APE, indicating better overall motion quality despite higher overall APE for some ablations.
- APE: Table 1 compares Average Positional Error (APE) in millimeters across the model, JL2P, Lin et al., and four ablations.
- AVE: The final model has the lowest Average Variance Error (AVE) for the root joint and the mean of all joints with and without global trajectory.
- CEE and SEE: The final model has the lowest Content Encoding Error (CEE) and Style Encoding Error (SEE), for which lower values are better.