Source-linked AI summary
Text2Gestures: A Transformer-Based Network for Generating Emotive Body Gestures for Virtual Agents
Uttaran Bhattacharya, Nicholas Rewkowski, Abhishek Banerjee, Pooja Guhan, Aniket Bera, Dinesh Manocha
TL;DR
Text2Gestures addresses the challenge of generating emotionally expressive, full-body gestures aligned with natural-language text for virtual-agent narration and conversation. It uses an end-to-end transformer conditioned on affective features, task, emotion, gender, and handedness, achieving state-of-the-art text-aligned gesture generation and positive user-study results. Around 91% of respondents found the gestures at least plausible, with perceived and intended emotions showing a minimum Pearson coefficient of 0.77 in valence.
Problem
Real-time virtual-agent interactions require gestures that adapt to associated text and social context, but manually designed, pre-generated, and speech-dependent approaches are limited.
Method
An end-to-end transformer generates full-body 3D gesture pose sequences from text while conditioning on affective features, acting task, intended emotion, gender, and handedness.
Results
The network achieved state-of-the-art text-aligned gesture generation, ran at 312.5 fps on an Nvidia GeForce GTX 1080Ti GPU, and received positive plausibility and emotion-perception evaluations.
Takeaways & Limitations
Text2Gestures supports interactive generation of emotionally expressive full-body gestures aligned with text for virtual-agent narration and conversation.
Takeaways & Limitations
The system maps complete sentences to gestures that return to an end-of-sentence pose, so it does not exploit continuity between adjacent sentences.
Abstract
from arXiv · showhide
We present Text2Gestures, a transformer-based learning method to interactively generate emotive full-body gestures for virtual agents aligned with natural language text inputs. Our method generates emotionally expressive gestures by utilizing the relevant biomechanical features for body expressions, also known as affective features. We also consider the intended task corresponding to the text and the target virtual agents' intended gender and handedness in our generation pipeline. We train and evaluate our network on the MPI Emotional Body Expressions Database and observe that our network produces state-of-the-art performance in generating gestures for virtual agents aligned with the text for narration or conversation. Our network can generate these gestures at interactive rates on a commodity GPU. We conduct a web-based user study and observe that around 91% of participants indicated our generated gestures to be at least plausible on a five-point Likert Scale. The emotions perceived by the participants from the gestures are also strongly positively correlated with the corresponding intended emotions, with a minimum Pearson coefficient of 0.77 in the valence dimension.
1 INTRODUCTION
Text2Gestures addresses the challenge of generating real-time body gestures aligned with natural-language text while incorporating emotional and agent-specific factors. It uses a transformer-based network and reports interactive generation, state-of-the-art text alignment, and positive user evaluations.
- Motivation: Interactive text-to-gesture generation is needed because pre-generated or manually designed animations cannot adapt to words and social context in real time.Speech-based approaches also require prerecorded speech, whereas text-based mapping avoids that production requirement.
- Motivation: Emotional body expressions contribute to virtual-agent realism and provide cues for perceived emotion.Relevant biomechanical properties, called affective features, include arm-swing rate, stride length, shoulder and spine posture, and head jerks.
- Approach: The proposed transformer network generates 3D full-body pose sequences from one text sentence at a time for narration or conversation.The system is designed for interactive applications involving virtual agents and human participants.
- Approach: The generation process conditions gestures on acting task, intended emotion, gender, and handedness.These conditions support emotionally expressive gestures tailored to the virtual agent and interaction setting.
- Results: 312.5 fps generation was achieved on an Nvidia GeForce GTX 1080Ti GPU, while the network showed state-of-the-art performance against ground-truth motion-capture sequences.The evaluation used the MPI Emotional Body Expressions Database.
- Results: Around 91% of respondents rated the generated gestures at least plausible, and perceived and intended emotions had a minimum Pearson coefficient of 0.77 in valence.The web study collected 600 total responses.
2 RELATED WORK
Related work covers emotion perception from body movement, gesture generation aligned with speech or text, and broader stylistic motion synthesis. Text2Gestures differs by producing emotionally expressive, semantic-aware full-body gestures directly from text at interactive rates.
- Generating Gestures: Rule-based gesture systems are limited because their gesture range is confined to a designed set.The paper therefore emphasizes data-driven approaches for broader gesture generation.
- Perceiving Emotions from Body Expressions: Body expressions provide emotion cues and can disambiguate emotions that facial or vocal cues leave ambiguous.Researchers identify biomechanical affective features such as rapid arm swings, expanded upper body, and slouching shoulders.
- Generating Gestures Aligned with Speech and Text: Prior systems commonly learn gesture mappings from speech signals, while text-based methods instead use language transcripts as input.Text input avoids the need for prerecorded human or machine speech.
- Generating Gestures Aligned with Speech and Text: Text2Gestures maps text to semantic-aware, human-like full-body 3D pose sequences with an end-to-end transformer and emotional expressiveness.This contrasts with prior work using upper-body, PCA-based features and heuristic mapping to an NAO robot.
- Generating Stylistic Human Body Motions: The paper targets emotionally expressive gestures corresponding to text sentences at interactive rates, rather than locomotion, head-motion, or facial-muscle motion spaces.Related head-motion methods had not been extended to full-body motions.
- Generating Stylistic Human Body Motions: The directed pose graph represents the body as a tree of 23 joints, with the root joint at the root and head, wrists, and toes as leaf end-effectors.Appropriate joints are manipulated to generate emotive gestures.
3 TRANSFORMING TEXT TO GESTURES
The task is to generate a virtual agent’s body-gesture sequence from a natural-language sentence while conditioning on interaction context, emotion, and agent attributes.
- Problem Formulation: Given text words, an acting task, intended emotion, gender, and handedness, the model generates relative 3D joint rotations for the corresponding body gestures.The acting task is narration or conversation.
Q Prob[Q|W ;A,E,G,H]. (1)
Text2Gestures formulates text-to-gesture generation as conditional sequence transduction, representing text with embeddings and gestures with joint rotations. Its transformer encoder-decoder incorporates acting task, intended emotion, gender, and handedness while generating gestures causally from past history.
- Representing Inputs and Gestures: Text2Gestures represents each input word with a 300-dimensional GloVe embedding and each gesture pose with quaternion rotations relative to parent joints.The gesture sequence uses flattened unit-quaternion vectors across all body joints; quaternions avoid gimbal lock.
- Conditional Attributes: The model conditions gesture generation on acting task, intended emotion, agent gender, and agent handedness.Narration uses more exaggerated gestures than conversation, while intended emotions are mapped from categorical labels into VAD space.
- Transformer Encoder: A transformer encoder converts word embeddings into latent features, appends agent attributes, and maps the combined representation through fully connected layers.The encoder uses multi-head self-attention to model dependencies across input positions before producing the encoded representation.
- Transformer Decoder: The transformer decoder uses masked self-attention over gesture history and unmasked attention to encoded text features to predict subsequent gesture poses causally.Masking restricts attention to preceding target elements, enabling test-time generation when future gestures are unknown.
4 TRAINING THE TRANSFORMER-BASED NETWORK
The network is trained with losses that combine smooth rotational motion, pose-trajectory consistency, and emotionally expressive affective features. Its transformer architecture predicts valid rotations while conditioning gesture generation on emotion-related body characteristics.
- Network architecture: The transformer uses two encoder and decoder blocks with two attention heads, maps features to 200-dimensional outputs, and normalizes decoder predictions as valid rotations.Word embeddings come from pre-trained GloVe representations.
- Angle loss: The angle loss compares predicted and ground-truth joint rotations and their successive derivatives to encourage accurate and smooth motions.Rotations are represented as unit quaternions, while the loss uses squared Euler-angle differences modulo π.
- Pose loss: The pose loss adds squared differences between predicted and ground-truth joint positions computed through forward kinematics, constraining joint trajectories beyond absolute rotations.The loss uses each joint’s relative rotation and offset from its parent to compute positions.
- Affective loss: Figure 3 links high arousal with rapid limb movements and high dominance with expanded upper-body postures, while low values correspond to slower or contracted gestures.The algorithm uses these gesture characteristics to generate appropriate gestures.
- Affective loss: The affective loss penalizes differences between affective features extracted from predicted and ground-truth poses to preserve emotional expressiveness.The features use scale-independent angles, distance ratios, and area ratios, computed from the moving upper-body joints.
5 RESULTS
The evaluation compares Text2Gestures with prior text-to-gesture generation and ablated loss configurations, while also measuring interactive generation performance and VR integration.
- 5.1 Database: The method is evaluated on the MPI emotional body expressions database, whose sequences represent narration, scenario gesticulation, or conversation tasks with emotion, gender, and handedness annotations.The database contains 1,447 motion-captured sequences captured at 120 fps, lasting 4–20 seconds.
- 5.2 Training and Evaluation Routines: The comparison retrains Yoon et al.’s network on the same database and split, using PCA-reduced upper-body features and evaluating the same eight joints.Mean pose error is computed over all joints and time steps relative to each sequence’s normalized bounding-box diagonal.
- 5.3 Comparative Performance: 97%: Text2Gestures reduces mean pose error by around 97% compared with Yoon et al.’s method.The authors attribute the difference partly to Text2Gestures capturing high-amplitude oscillations that the lower-dimensional representation misses.
- 5.4 Ablation Studies: Removing angle or pose loss makes animations jerky, rigid, or constricted, while removing affective loss preserves fluid motion but makes emotional expressions inconsistent with intended emotions.The ablations remove each loss term individually and retrain the network from scratch.
- 5.5 Interfacing the VR Environment: 3.2 ms per frame: The system generates gesture animation files at an interactive rate of 312.5 frames per second on an Nvidia GeForce GTX 1080Ti GPU.The VR environment supports narration of selected stories and text-based conversational responses with associated gestures.
6 USER STUDY
The web study evaluates emotional perception and animation quality using generated and motion-captured gestures. Generated gestures received strong emotion correlations and were judged plausible at a rate comparable to motion capture, despite the gesture-only modality.
- 6.1 Procedure: The study measures both correlation between intended and perceived emotions and animation quality relative to original motion-captured sequences.It includes separate emotion-identification and naturalness-rating sections.
- 6.1 Procedure: Fifty participants completed the study, with 300 emotion responses and 150 naturalness responses for each of the generated and motion-captured conditions.Participants viewed randomly selected clips without being told which sequences were generated.
- 6.3.1 Correlation between Intended and Perceived Emotions: 0.77, 0.95, and 0.82: Pearson correlations between intended and perceived values were observed for valence, arousal, and dominance, respectively.All three dimensions showed strong positive correlations, with valence weakest and arousal strongest.
- 6.3.2 Quality of Gesture Animations: 91.33%: Generated gestures were rated plausible at least 86.67% of the time for motion-captured gestures under the study’s minimum Likert score of 3.Generated gestures received marginally higher proportions of ratings 4 and 5 than motion-captured gestures.
- 6.3.3 Miscellaneous Feedback: 14% of participants were distracted by missing facial emotions, 10% could not determine emotions from gestures alone, and 8% reported both difficulties.The virtual agents expressed emotions through gestures without facial or vocal modalities.
7 CONCLUSION
The method generates emotive 3D gesture sequences aligned with sentence-level natural language while accounting for task, emotion, gender, and handedness. User evaluations found the gestures broadly plausible and emotionally aligned with intended expressions.
- The method generates 3D pose sequences for emotive gestures aligned with one natural-language sentence at a time.
- It conditions gesture generation on narration or conversation, intended emotion, virtual-agent gender, and handedness.
- Generated gestures run in a few milliseconds on an Nvidia GeForce GTX 1080Ti GPU.
- A marginally higher percentage of participants rated generated gestures better than the original motion-captured sequences on virtual agents.
- Around 91% of respondents rated the generated gestures at least plausible on a five-point Likert scale.
- Across 600 responses from 50 participants, perceived and intended emotions showed a strong positive correlation, with a minimum Pearson coefficient of 0.77 in valence.
8 LIMITATIONS AND FUTURE WORK
The authors identify limitations in sentence-level gesture mapping, inter-sentence continuity, emotion representation, and the absence of facial or vocal expressiveness.
- Sentence-level text-to-gesture mapping cannot reveal how phrase-level gestures combine into gestures for full sentences.
- Because training samples begin from a rest pose, gestures return to the EoS pose after each sentence and do not model adjacent-gesture continuity.
- The network considers only the VAD representation for categorical emotion terms associated with texts.
- The evaluation excludes facial expressions and expressive voice tones, although these modalities can play dominant roles in conveying emotion.