Source-linked AI summary
BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis
Haiyang Liu, Zihao Zhu, Naoya Iwamoto, Yichen Peng, Zhengqing Li, You Zhou, Elif Bozkurt, Bo Zheng
TL;DR
Realistic, human-like conversational gesture synthesis lacks sufficiently large, richly paired datasets, suitable models, and standard semantic metrics. The paper introduces BEAT, CaMN, and SRGR; experiments report high data quality, valid evaluation metrics, and state-of-the-art baseline performance. BEAT is intended to support gesture synthesis and related cross-modality and emotional-motion research, within limitations of semantic annotations and controlled acting effects.
Problem
Realistic, vivid, human-like conversational gesture synthesis remains unsolved because available datasets, models, and standard evaluation metrics are limited.
Method
The paper builds BEAT, proposes the six-modality cascaded CaMN baseline, and introduces SRGR for evaluating semantic relevancy.
Results
Experiments demonstrate BEAT data quality, SRGR validity, and CaMN state-of-the-art performance.
Takeaways & Limitations
BEAT and its analyses may contribute to controllable gesture synthesis, cross-modality analysis, and emotional motion recognition.
Takeaways & Limitations
SRGR currently depends on semantic annotations, limiting its use for unlabelled datasets; the authors suggest a scoring network or semantic discriminator as a possible direction.
Abstract
from arXiv · showhide
Achieving realistic, vivid, and human-like synthesized conversational gestures conditioned on multi-modal data is still an unsolved problem due to the lack of available datasets, models and standard evaluation metrics. To address this, we build Body-Expression-Audio-Text dataset, BEAT, which has i) 76 hours, high-quality, multi-modal data captured from 30 speakers talking with eight different emotions and in four different languages, ii) 32 millions frame-level emotion and semantic relevance annotations. Our statistical analysis on BEAT demonstrates the correlation of conversational gestures with facial expressions, emotions, and semantics, in addition to the known correlation with audio, text, and speaker identity. Based on this observation, we propose a baseline model, Cascaded Motion Network (CaMN), which consists of above six modalities modeled in a cascaded architecture for gesture synthesis. To evaluate the semantic relevancy, we introduce a metric, Semantic Relevance Gesture Recall (SRGR). Qualitative and quantitative experiments demonstrate metrics' validness, ground truth data quality, and baseline's state-of-the-art performance. To the best of our knowledge, BEAT is the largest motion capture dataset for investigating human gestures, which may contribute to a number of different research fields, including controllable gesture synthesis, cross-modality analysis, and emotional gesture recognition. The data, code and model are available on https://pantomatrix.github.io/BEAT/.
1 Introduction
Realistic conversational gesture synthesis remains difficult because existing resources lack scale, paired modalities, speaker comparability, and emotion annotations. BEAT addresses these gaps with a large annotated motion-capture dataset, alongside CaMN and SRGR, with experiments supporting their reported quality and performance.
- Research gap: Existing gesture synthesis remains challenging because prior datasets and models provide limited scale, modality coverage, speaker-style comparability, and emotion annotation.Earlier approaches used limited motion-capture or pseudo-labeled data and often one or two modalities.
- BEAT dataset: 76 hours of motion-captured data from 30 speakers cover four languages, eight emotions, four modalities, and more than 2500 topic-segmented sequences.The dataset includes body, expression, audio, and text data with frame-level semantic and emotional annotations.
- BEAT dataset: BEAT uses controlled recording, professional feedback, and re-recording to support motion quality, expressiveness, and diversity.Speakers also contributed paired recordings across languages, durations, phonemes, and topics.
- Models and metric: CaMN synthesizes body and hand gestures from six modalities through cascaded encoders and decoders that enhance audio and facial contributions.Its inputs include audio, text, facial blendweights, speaker identity, emotion, and semantic score.
- Models and metric: SRGR evaluates semantic relevancy by weighting PCK according to ground-truth semantic scores, while experiments report valid metrics, quality data, and state-of-the-art CaMN performance.The paper describes BEAT as the largest motion-capture dataset for investigating human gestures, to the authors’ knowledge.
2 Related Work
Prior conversational gesture datasets span motion capture and pseudo-label approaches, but limited data and noisy ground truth constrain generalization and quality. Related gesture-generation baselines use text, audio, or combined audio-text conditioning with CNN, LSTM, and generative methods.
- Motion-capture datasets: Motion-capture datasets include 89-minute to 4-hour recordings, often with few speakers or narrow topics.Examples include Volkova et al., Takeuchi et al., and the single-speaker Trinity dataset.
- Pseudo-label datasets: Pseudo-label datasets scale to 33–144 hours by extracting poses from online videos, but their noisy ground truth creates a trade-off with generalization capability and quality.Examples include S2G, a 33-hour full-body dataset, and TED-based data reported at 97 hours.
- Semantic gesture analysis: Action-recognition and sign-language datasets do not directly apply because conversational gestures have more complex semantics than single actions.Conversational gestures occur in natural dialogue rather than isolated labeled actions.
- Gesture synthesis methods: Early gesture-synthesis baselines condition on text, audio, or audio-text inputs and use CNN or LSTM end-to-end models.Later work explores representation choices, adversarial training, and generative modeling techniques.
3 BEAT: Body-Expression-Audio-Text Dataset
BEAT is a controlled, multi-modal motion-capture dataset designed to study conversational gestures across speakers, languages, emotions, and semantic relevance. Its annotations and analyses expose relationships between gestures and facial expressions, emotions, semantics, audio, text, and speaker identity.
- Data acquisition: BEAT comprises motion, facial, audio, text, emotion, and semantic-relevance information collected in conversation and self-talk sessions.The dataset uses synchronized motion capture, facial blendshape capture, audio recording, and annotated textual and affective information.
- Emotion and gesture categories: Eight emotions are represented, while conversation gestures are categorized as talking, instantaneous reactions, thinking, and asking.Self-talk recordings use neutral, anger, happiness, fear, disgust, sadness, contempt, and surprise.
- Subject and language distribution: 76 hours of data from 30 speakers cover four languages, with controlled language and accent proportions.The corpus includes 60h English, 12h Chinese, 2h Spanish, and Japanese data, with English recordings split between native and fluent non-native speakers.
- Multi-modal analysis: Gestures show emotion-specific characteristics, including larger and faster movements for speaker-2 when angry than when happy.T-SNE visualization separates gestures across emotion categories, while examples illustrate within-speaker stylistic differences.
- Multi-modal analysis: 83% of gestures have low semantic scores of ≤0.2, and semantic relevance varies with speaker, word, frequency, position, and content.The analysis identifies substantial randomness in which gestures accompany the same uttered text.
4 Multi-Modal Conditioned Gestures Synthesis Baseline
CaMN is a cascaded multi-modal baseline that encodes text, emotion, speaker identity, audio, and facial expression before reconstructing body and hand gestures. It combines reconstruction and adversarial objectives, modulated by semantic relevance to emphasize spatial agreement for semantically relevant gestures.
- Architecture: CaMN encodes text, emotion, speaker identity, audio, and facial blendshape weights in a multi-stage cascade for gesture synthesis.The architecture is designed to enhance the contribution of audio and facial modalities by concatenating them with previously encoded features.
- Text Encoder: The text encoder applies an 8-layer temporal convolutional network to a 2f = 34-frame context and produces zT ∈R128.Words are first represented with 300-dimensional FastText embeddings and then refined by the temporal encoder.
- Condition encoders: Speaker identity uses an embedding layer, while emotion combines an embedding layer with a 4-layer temporal convolutional network.Speaker identity is taken from the current frame, whereas emotion features model temporal variation.
- Audio and facial encoders: Audio and facial encoders fuse features from preceding modalities through temporal convolutional and MLP layers.The audio encoder uses 12 TCN layers and a 2-layer MLP, while the facial encoder uses 8 TCN layers and a 2-layer MLP.
- Body and Hands Decoders: Cascaded LSTM-plus-MLP decoders reconstruct body gestures first and hands gestures using body information, producing 27×3 body and 48×3 hand outputs.The decoders combine five modality features with previous gestures, using a four-frame seed pose.
- Training objective: Training combines body-and-hand L1 reconstruction with adversarial loss, weighting them by semantic relevance.Higher semantic relevance strengthens the reconstruction penalty and decreases the adversarial penalty.
5 Metric for Semantic Relevancy
SRGR evaluates semantic relevance by weighting joint-recall accuracy according to ground-truth semantic scores. It is intended to reflect whether generated gestures recall semantically relevant motion while remaining vivid and diverse.
- Metric definition: SRGR weights Probability of Correct Keypoint by the semantic scores of ground-truth gestures.PCK counts joints successfully recalled within a specified threshold δ.
- Metric definition: SRGR averages thresholded joint recall across T frames and J joints, scaled by semantic relevance λ.The indicator function counts joints whose generated and ground-truth positions fall within δ.
- Metric motivation: The metric emphasizes recalling gestures in clips of interest and is presented as more aligned with perceived valid gesture diversity than L1 variance.This interpretation connects semantic weighting to the evaluation of vividness and diversity.
6 Experiments
The experiments evaluate SRGR, BEAT’s data quality, CaMN and modality contributions through subjective and objective comparisons. Results support SRGR’s perceptual validity, BEAT’s stronger data-quality preferences, and state-of-the-art performance for CaMN and its cascaded design.
- 6.1 Validness of SRGR: 160 participants evaluated 200 gesture clips, finding SRGR more consistent with human perception and less variable than L1 diversity.Clips covered Seq2Seq, S2G, A2G, MultiContext, and ground-truth results, with diversity and attractiveness rated on five-point Likert scales.
- 6.2 Data quality: 120 participants preferred BEAT over Trinity across gesture correctness, diversity, and gesture-audio synchrony, especially for hand movements.The comparison separately evaluated body and hands using clips sampled from ground-truth and generated results.
- 6.3 Baseline evaluation: CaMN performed best in FGD, SRGR, and BeatAlign among methods trained on BEAT, while both end-to-end and cascaded models achieved state-of-the-art performance across metrics.The compared methods included S2G, A2G, Seq2Seq, and MultiContext.
- 6.4 Ablation study: The cascaded connection outperformed the end-to-end approach by using prior human knowledge to extract features from different modalities.This comparison is reported in the ablation study.
- 6.4 Ablation study: Removing audio significantly reduced synchrony, while speaker identity had little effect; removing weighted semantic loss improved synchrony.Text alignment, padding, lip motion, and emotion retained or influenced some synchronization.
- 6.5 Limitation: SRGR depends on semantic annotations, limiting its use for unlabelled datasets; a scoring network or semantic discriminator is suggested as a possible direction.This limitation is stated explicitly in the paper’s discussion.
7 Conclusion
The paper presents BEAT, a large-scale multimodal dataset with semantic and emotional annotations, alongside a six-modality cascade-based gesture-synthesis baseline and SRGR evaluation metric. The authors report state-of-the-art performance and potential relevance to several gesture-related research fields.
- 7 Conclusion: BEAT is a large-scale, high-quality dataset with semantic and emotional annotations for generating more human-like conversational gestures.The conclusion identifies the dataset as a central contribution.
- 7 Conclusion: The cascade-based baseline uses six modalities for gesture synthesis and achieves state-of-the-art performance.The conclusion pairs this result with the dataset contribution.
- 7 Conclusion: SRGR is introduced to evaluate semantic relevancy in conversational gestures.The conclusion identifies SRGR as the paper’s evaluation contribution.
- 7 Conclusion: The dataset and statistical experiments may benefit controllable gesture synthesis, cross-modality analysis, and emotional motion recognition.These are the research fields named by the authors.
A Annotation Interface and Measurement of Agreement.
The annotation workflow combines segment- and keyword-level judgments with text alignment to derive frame-level semantic relevance, while emotion annotations show high inter-rater agreement. Separate annotator groups handled emotion and semantic tasks.
- Annotation interface: Two annotators label emotion segment start and end times, while a different annotator group handles semantic annotations.The paper reports separate groups for the two annotation tasks.
- Measurement of agreement: Frame-level semantic relevance equals the gesture-segment semantic score multiplied by the keyword semantic score.The two annotation levels can also be used separately.
- Annotation interface: Annotators label gesture segments and keywords, and post-processing combines these labels with text alignment to produce frame-level semantic relevance scores.The interface supports both emotion and semantic annotation tasks.
- Measurement of agreement: 96% agreement was achieved across approximately 16M emotion-annotated frames.Agreement counted cases where the two annotators assigned the same label.
- Measurement of agreement: Emotion annotation duration and clip-level annotation counts are summarized in Figure 8.The figure reports total annotated duration in seconds and the distribution of annotation counts per clip.
B Details of Text Content and Speaker Information.
BEAT’s supplementary materials document its conversation topics, speaker diversity and style controls, released multimodal file formats, and avatar, skeleton, marker, and facial representation details. The corpus also targets phonetic coverage and includes body, hand, and facial assets.
- Text content: Conversation content covers daily topics, while vowel and consonant distributions are broadly consistent with frequently used 3000-word vocabulary.The corpus includes ten debate topics, ten introduction topics, and 120 Self-Talk answers.
- Speaker information: Speakers were selected across countries, gender, ages, and ethnicity to investigate style differences.The dataset records speaker information including gender, originating country, language status, duration, age, and ethnicity.
- Speaker information: The authors filtered out about 21 hours of data and six speakers because of inconsistent styles, while observing different gestures in Self-Talk and Conversation sessions.Some speakers gestured frequently in conversation but showed almost no gestures in Self-Talk.
- Released data: Released files include body and hand motion, stereo audio, facial blendshape weights, facial meshes for eight speakers, text-audio alignment, and semantic and emotion annotations.The listed formats include BVH, WAV, JSON, FBX, TextGrid, and text files.
- Motion and facial representation: Facial expressions use FACS-based blendshapes, with expression targets activating facial regions such as the mouth, eyes, or eyebrows.Facial meshes and avatars were prepared for visualization and demonstration.
D Additional Discussions for SRGR, FGD and BeatAlign.
The discussion compares SRGR with other gesture-evaluation metrics and examines their alignment with human perception, physical correctness, and audio-gesture synchrony.
- SRGR is closer to human perception of gesture diversity and attractiveness than equal-weight L1 diversity.
- FGD depends on gesture feature representations and may miss obvious jitters in synthesized sequences.
- 6% higher BeatAlign scores than ground truth appeared in 300 sampled clips, while conversational-gesture evaluation achieved 83% average precision.
- BeatAlign uses gesture and audio beat distances to measure gesture-audio beat similarity.
E More Subjective Results and Videos.
The paper provides subjective visualizations comparing ground-truth gestures with generated gestures under neutral and fear emotion conditions, including results in Japanese.
- Additional subjective results include gesture videos and examples in other languages, including Japanese.
- Figure 15 compares ground-truth gestures with generated results conditioned on neutral and fear emotions.
F Details of baseline training
Baseline training uses speaker-specific train, validation, and test splits, with model selection based on validation FGD.
- Each speaker’s data is divided into training, validation, and test sets using a 10:1:1 ratio.
- Baseline models select the best epoch by the lowest validation FGD score.
- Table 10 lists the best epoch selected for each baseline.