Source-linked AI summary
Language2Pose: Natural Language Grounded Pose Forecasting
Chaitanya Ahuja, Louis-Philippe Morency
TL;DR
Language-to-pose generation requires mapping linguistic descriptions of motion into coherent animations across language and pose modalities. JL2P learns a joint embedding end to end with curriculum learning, emphasizing shorter and easier sequences before longer and harder ones. Objective and human evaluations report more accurate and visually representative animations than other data-driven approaches.
Problem
Language-to-pose generation must map language concepts such as actions, speed, direction, and targets to coherent motion animations across different modalities.
Method
JL2P jointly embeds language and pose end to end, using curriculum learning that progresses from shorter, easier sequences to longer, harder ones.
Results
JL2P generates more accurate animations, with at least 9% improvement over Lin et al. for all joints and 75% human preference.
Takeaways & Limitations
Objective metrics and human judgments support JL2P as a more accurate and visually representative language-to-pose approach than other data-driven models.
Takeaways & Limitations
The reported baseline comparison used an independent implementation and training because code or pretrained models for the referenced work were unavailable.
Abstract
from arXiv · showhide
Generating animations from natural language sentences finds its applications in a a number of domains such as movie script visualization, virtual human animation and, robot motion planning. These sentences can describe different kinds of actions, speeds and direction of these actions, and possibly a target destination. The core modeling challenge in this language-to-pose application is how to map linguistic concepts to motion animations. In this paper, we address this multimodal problem by introducing a neural architecture called Joint Language to Pose (or JL2P), which learns a joint embedding of language and pose. This joint embedding space is learned end-to-end using a curriculum learning approach which emphasizes shorter and easier sequences first before moving to longer and harder ones. We evaluate our proposed model on a publicly available corpus of 3D pose data and human-annotated sentences. Both objective metrics and human judgment evaluation confirm that our proposed approach is able to generate more accurate animations and are deemed visually more representative by humans than other data driven approaches.
1. Introduction
Language-to-pose generation must connect language and pose across modalities while translating linguistic descriptions into coherent animations. JL2P addresses this with joint embedding and curriculum learning.
- Language-to-pose generation supports applications including movie script visualization, virtual human animation, and robot motion planning.
- The model must map language and pose into a joint space and decode animations from that representation.
- Verbs, adverbs, nouns, and adjectives convey actions, speed, locations, and directions that must be mapped to pose sequences.
- JL2P learns a joint language-pose embedding and trains with shorter, easier sequences before longer, harder ones.The model also uses Smooth L1 as its distance metric to improve robustness to dataset outliers.
- Objective metrics and human evaluations assess whether generated animations are accurate and natural relative to other data-driven models.
2. Related Work
Prior pose-forecasting approaches commonly condition on actions, pose history, images, speech, or low-level controls. Language offers finer-grained control over action, speed, acceleration, direction, and targets, motivating JL2P’s jointly learned embedding.
- Pose forecasting models generate future poses from pose history, with long-term methods additionally modeling joint positions and trajectories.
- Existing approaches may condition on actions, pose history, terrain, trajectories, or task-specific footstep plans.
- The paper replaces low-level control parameters with natural language to control generated actions, speed, and direction.
- Image- and speech-conditioned methods provide context but do not offer fine-grained control over movement speed, acceleration, or next movements.
- Natural language combines verbs, adverbs, nouns, and adjectives to specify actions, speed, acceleration, direction, or targets.
- Video-based language-to-pose work can be limited by restricted translation and occluded lower bodies, while Lin et al. learn language mappings after pose-space pretraining.
- JL2P instead learns language and pose embeddings jointly with curriculum learning.
3. Problem Statement
The task translates a sentence into a semantically coherent sequence of 3D poses. It is formulated as learning a trainable mapping from word-vector sequences to pose sequences.
- A sentence such as “A person walks in a circle” should produce an animation combining circular motion with walking.
- The input is an N-word sequence X1:N, and the output is a T-step sequence of 3D poses Y1:T coherent with sentence semantics.
- Each word vector x_i has dimension K, while each pose y_t is a J×3 matrix whose rows are joints and columns are xyz coordinates.
- The model f maps an input space of K-dimensional word sequences to an output space of J×3×T pose sequences.
- Θ denotes the trainable parameters of the language-to-pose model.
4. Joint Language-to-Pose
JL2P learns a shared language–pose embedding and trains it end-to-end with a curriculum that progresses from short, easy predictions to longer, harder sequences. A joint decoder and translation losses connect language inputs to generated pose animations.
- Joint embedding: JL2P maps sentences and pose sequences into a shared latent embedding space so they represent the same concept.The sentence and pose encoders estimate embeddings z_x and z_y, which are trained to lie close together.
- Translation losses: A pose decoder generates animations from either language or pose embeddings, with outputs trained toward the target pose sequence.The cross-modal loss uses language as input and poses as outputs, while the uni-modal loss reconstructs poses.
- Joint optimization: The joint loss combines cross-modal translation and autoencoder losses to align the modalities while training the pose decoder.Joint optimization is described as improving embedding alignment and supporting inference from the shared space.
- Curriculum learning: Curriculum learning begins with two predicted time steps and doubles the prediction length as validation performance permits.The stages progress from short motions such as walking-leg or waving-hand movements toward the maximum prediction horizon.
- Optimization: Smooth L1 is used as the pose distance because it is less outlier-sensitive than L2 and smoother near zero than L1.The paper describes Smooth L1 as more stable than L1 while contrasting it with the L2 baseline.
5. Experiments
The experiments evaluate language-grounded pose forecasting through prediction accuracy, human judgments, and the modeling of nuanced language concepts. They use the KIT Motion-Language Dataset, objective pose metrics, subjective pairwise video comparisons, and ablations of JL2P’s main components.
- Evaluation goals: The experiments test prediction accuracy, human judgment, and whether generated poses capture speed, direction, and action.These correspond to the paper’s three stated evaluation challenges.
- Dataset: KIT Motion-Language Dataset contains 3911 motion recordings and 6278 human-annotated English descriptions, with 20% held out for evaluation.The motions span locomotion, performing, and gesticulation, including descriptions of speed, direction, and repetition.
- Preprocessing: Motion preprocessing canonicalizes orientation, represents local joint positions and root motion, and subsamples sequences from 100 Hz to 12.5 Hz.The sampling rate is chosen to support decoder regression without compromising perceived animation quality.
- Models and baselines: JL2P uses GRUs for pose encoding and decoding, with a residual decoder connection, while its language encoder uses LSTMs over pretrained Word2Vec token embeddings.A BERT language-encoder variant did not show significant improvements.
- Models and baselines: The study compares JL2P with Lin et al. and ablations removing curriculum learning, Smooth L1, or joint embedding training.The Lin et al. comparison uses an independent implementation trained on the same data.
- Objective evaluation: Table 1 reports Average Position Error across joints, with lower values indicating better performance.The caption states that JL2P and its variants consistently improve accuracy over Lin et al. across all joints.
- Subjective evaluation: The user study asks annotators to choose which of two videos better matches the same input sentence, rejecting annotations completed in under 20 seconds.Comparisons include Lin et al., ground truth, JL2P, and JL2P ablations.
6. Results and Discussion
JL2P improves objective pose accuracy and human preference over Lin et al., with curriculum learning and joint embedding contributing to performance. Qualitative examples show that it captures language-defined direction, speed, and actions, although Root trajectory prediction remains difficult over time.
- Prediction Accuracy by Joint Space: At least 9% improvement over Lin et al. is reported for all joints, with a maximum improvement of around 15% for Root.Improved long-term Root prediction can reduce foot sliding when translation outpaces foot-motion frequency.
- Prediction Accuracy by Joint Space: Curriculum learning benefits pose generation, while replacing Smooth L1 with L2 lowers APE by 4%.Removing the joint embedding loss lowers APE by only 1%, whereas the curriculum first predicts shorter sequences before longer ones.
- Prediction Accuracy by Joint Space: Root APE increases fastest over time, because velocity-based Root coordinates accumulate integration errors; the error rises sharply after about 500ms.The plotted joints are Root, Legs, Arms, Torso, and Head, and lower APE is better.
- Prediction Accuracy by Joint Space: JL2P consistently improves PCK over its ablations and Lin et al. across thresholds σ from 35 to 55.Higher PCK values indicate better keypoint accuracy.
- Human Judgment: JL2P receives 75% human preference against Lin et al., while preference scores consistently decline for its variants.Ground truth is preferred by 10 percentage points more than JL2P.
- Human Judgment: Removing joint embedding reduces preference to 60% despite little change in objective scores, showing that objective metrics alone do not capture animation quality.The study therefore supports combining human judgment with objective metrics when evaluating pose generation.
- Modeling nuanced language concepts: Qualitative examples show that JL2P models direction, speed, and diverse actions such as kneeling and jogging in generated animations.Its trajectories resemble ground truth more closely than Lin et al. for direction and speed examples.
7. Conclusions
JL2P integrates language and pose in an end-to-end joint embedding and uses curriculum learning to generate animations from descriptions. On paired 3D pose and sentence data, it achieved stronger objective accuracy and human-rated visual representation than the state-of-the-art model.
- JL2P integrates language and pose into an end-to-end joint embedding for generating animations conditioned on descriptions.
- Curriculum learning trains the model on shorter sequences before progressively introducing longer ones.
- The model was evaluated on paired 3D pose data and human-annotated sentences using objective metrics and a user study.
- JL2P generated more accurate animations and received stronger human visual-representation judgments than the state-of-the-art model.