Source-linked AI summary
EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling
Haiyang Liu, Zihao Zhu, Giorgio Becherini, Yichen Peng, Mingyang Su, You Zhou, Xuefei Zhe, Naoya Iwamoto, Bo Zheng, Michael J. Black
TL;DR
Holistic co-speech gesture generation is limited by gaps in comprehensive data and models. The paper introduces BEAT2 and EMAGE, which generate gestures from audio and partial inputs; it reports state-of-the-art body and face generation with four seed frames.
Problem
Holistic co-speech generation faces limited availability of comprehensive data and models, while existing datasets vary in formats and evaluation metrics across subareas.
Source: §1 · pp. 1–2Method
The paper introduces BEAT2, a unified SMPL-X and FLAME mesh-level dataset, and EMAGE, which reconstructs masked gesture latents and decodes holistic motion from audio and partial gestures.
Source: §1 · p. 2Results
EMAGE reports state-of-the-art body and face gesture generation using only four seed frames.
Source: §1 · p. 2Takeaways & Limitations
EMAGE can incorporate non-holistic datasets such as Trinity and AMASS, whose joint training improves performance on the BEATv1.3 test set.
Source: §5.2.3 · p. 8
Abstract
from arXiv · showhide
We propose EMAGE, a framework to generate full-body human gestures from audio and masked gestures, encompassing facial, local body, hands, and global movements. To achieve this, we first introduce BEAT2 (BEAT-SMPLX-FLAME), a new mesh-level holistic co-speech dataset. BEAT2 combines a MoShed SMPL-X body with FLAME head parameters and further refines the modeling of head, neck, and finger movements, offering a community-standardized, high-quality 3D motion captured dataset. EMAGE leverages masked body gesture priors during training to boost inference performance. It involves a Masked Audio Gesture Transformer, facilitating joint training on audio-to-gesture generation and masked gesture reconstruction to effectively encode audio and body gesture hints. Encoded body hints from masked gestures are then separately employed to generate facial and body movements. Moreover, EMAGE adaptively merges speech features from the audio's rhythm and content and utilizes four compositional VQ-VAEs to enhance the results' fidelity and diversity. Experiments demonstrate that EMAGE generates holistic gestures with state-of-the-art performance and is flexible in accepting predefined spatial-temporal gesture inputs, generating complete, audio-synchronized results. Our code and dataset are available https://pantomatrix.github.io/EMAGE/
1. Introduction
Holistic co-speech gesture generation remains constrained by limited comprehensive data and models. The paper introduces BEAT2 and EMAGE to support unified training and audio-synchronized generation from audio and partial gesture inputs.
1. Introduction
- The framework targets a gap left by face-, body-, and hand-focused models and a few merged models: limited comprehensive data and models for full-body co-speech generation.It accepts partial spatial-temporal predefined gestures and completes remaining frames in synchronization with audio.Source: §1 · p. 1
- BEAT2 unifies mesh-level data with MoShed SMPL-X body and FLAME head parameters for training and evaluating holistic gestures across subareas.BEAT uses ARKit blendshape weights and Vicon skeletons without mesh representation, complicating shared training and vertex-based evaluation.Source: §1 · p. 2
- EMAGE reconstructs gesture latents from masked body joints using switchable gesture self-attention and audio-gesture cross-attention, then decodes local face and body gestures and global translation.Four compositional pretrained VQ-VAEs decode local gestures, while a pretrained Global Motion Predictor decodes global translations.Source: §1 · p. 2
- EMAGE generates coherent body and face gestures with audio and partial gesture priors, and reports state-of-the-art performance using only four seed frames.It also incorporates additional non-holistic gesture datasets for training.Source: §1 · p. 2
2. Related Work
Prior co-speech animation datasets and models often address separate body or face outputs, with differences in capture type, representation, and motion scope. The related work positions BEAT2 as a holistic mesh-level dataset and EMAGE as a model accepting masked gestures for full-body generation.
2. Related Work
- The related work distinguishes pseudo-ground-truth datasets, whose accuracy is limited, from motion-captured datasets, and notes that face and body are often handled separately.It reports an average 33.4 mm error for a state-of-the-art monocular 3D pose estimator on Human3.6M versus 0.142 mm for Vicon motion capture.Source: §2 · p. 2
- BEAT2 is described as the largest motion-captured dataset with holistic, academic community-standard mesh-level information.Earlier datasets vary in capture type and representation; BEAT includes body pose and facial blendshape weights but lacks mesh data.Source: §2 · p. 2, Table 1 · p. 3
- Earlier models address face or body motion, while EMAGE is presented as the first to accept audio with partially or completely masked gestures and generate full-body audio-synchronized results.TalkSHOW lacks lower-body and global motion, shared body-face information, and support for partially masked body hints.Source: Table 2 · p. 3, §2 · p. 3
3. BEAT2
BEAT2 converts BEAT’s skeleton and facial blendshape data into unified SMPL-X body and FLAME head parameters. Its construction combines marker-based body fitting, physical-rule refinement, and an optimized mapping from ARKit blendshapes to FLAME.
3. BEAT2
- BEAT2 derives mesh-level SMPL-X and FLAME parameters from BEAT, whose Vicon skeleton and ARKit facial data lack mesh-level shapes and vertices.BEAT uses 78-joint BVH motion files and R51 facial blendshape weights.Source: §3 · p. 3
3.1. Body Parameters Initialization via MoSh++
- The MoSh++ initialization fits body shape, pose, and translation to captured markers, then optimizes framewise poses and translations using weighted losses for accuracy and plausibility.The fitting first selects 12 frames and fixes marker offsets and body shape; loss terms include data fit, surface distance, regularization, priors, velocity constancy, and soft tissue.Source: §3.1 · p. 3
3.2. Body Parameters Refinement
- MoSh++ body parameters are refined with hard-coded rules for head-and-neck proportions, finger bending, and outlier smoothing.The head and neck should approximate one-seventh of total body length; non-thumb fingers should not bend backward, and values beyond 3σ are adjusted and blended with adjacent 10 frames.Source: §3.2 · p. 4
3.3. Blendshape Weights to FLAME Parameters
- ARKit’s 51 blendshape weights are mapped to 103 FLAME parameters using handcrafted FLAME templates, addressing topology differences that make direct wrapped-vertex optimization unsatisfactory.The FLAME parameters comprise 100 expression dimensions and 3 jaw-movement dimensions; the mapping is optimized to reduce vertex error.Source: §3.3 · p. 4
4. EMAGE
EMAGE jointly trains masked-gesture reconstruction and audio-conditioned generation to use body hints for holistic gesture decoding. Its design combines separated gesture priors, adaptive speech-feature fusion, and segment-specific decoding.
4. EMAGE
- EMAGE jointly optimizes masked gesture reconstruction and audio-conditioned generation, using masked body hints and distinct decoding paths for face, local body, and global translation.The framework reconstructs gesture latents with a pretrained VQ-Decoder and estimates global translations with a pretrained Global Motion Predictor.Source: §4 · p. 4, Figure 3 · p. 5
4.1. Compositional Discrete Face and Body Prior
- Four VQ-VAEs separately encode face, upper body, hands, and lower body to reduce the risk that a single model overlooks less frequent movements.The paper notes that a model trained on combined upper- and lower-body motion may neglect elbow movements when a speaker is constantly walking.Source: §4.1 · p. 5
4.2. Content and Rhythm Self-Attention
- Content Rhythm Attention adaptively merges onset-and-amplitude rhythm features with transcript word embeddings, using separate encoders for face and body.The adaptive fusion favors content or rhythm at different frames.Source: §4.2 · pp. 5–6
4.3. Masked Audio-Conditioned Gesture Modeling
- Masked gesture reconstruction encodes body hints; the generation path selectively fuses those hints with audio features to reconstruct gesture latents.Training masks spatial joints and temporal frames at a ratio increased from 0 to 95% across epochs, and optimizes latent-code classification and reconstruction losses to encourage diverse results.Source: §4.3 · p. 6
- BEAT2 outperforms the PGT dataset [65] for both body and face, performs slightly better on body than mocap [46], and lower on face than 3D scan-based datasets [14].These are user-preference win-rate comparisons.Source: Table 3 · p. 6
4.4. Face and Translation Decoding
- For facial latent decoding, EMAGE concatenates masked body hints with audio features; for global translation, it predicts from decoded local lower-body motion.The pretrained Global Motion Predictor is reported to significantly reduce foot sliding.Source: §4.4 · p. 6
5. Experiments
Experiments evaluate BEAT2’s dataset quality and EMAGE’s holistic gesture generation. They report strong results with four-frame seed poses, benefits from masked gesture modeling, and improved performance from combining datasets.
5.1. Dataset Quality Evaluation
- BEAT2 contains 60 hours after low-finger-quality data are removed, split into BEAT2-Standard (27 hours) and BEAT2-Additional (33 hours).The perceptual comparison samples equal-duration sequence pairs and asks participants to judge upper-body quality, matching TalkSHOW’s scope.Source: §5 · p. 6, §5.1 · pp. 6–7
5.2. Model Ability Evaluation
- EMAGE produces diverse, semantic-aware, audio-synchronized gestures and accepts visible body hints on selected frames and joints to guide or customize motion.Examples include raising both hands for “spare time,” relaxing gestures for “hike in nature,” and changing walking orientation.Source: Figure 6 · p. 7
5.2.1 Quantitative and Qualitative Analysis
- With four-frame seed poses, EMAGE outperforms prior state-of-the-art methods; user preference is 14% higher for body and 23% for face gestures.The quantitative evaluation reports improved body FGD with masked gesture modeling.Source: Table 5 · p. 7, Table 4 · p. 7, §5.2.1 · p. 8
5.2.2 Ablation Analysis
- Ablations report benefits from separate VQ-VAEs, adaptive rhythm-content fusion, and masked body hints.Adaptive fusion improves FGD and alignment; masked hints improve all objective metrics and reduce the likelihood of incorrect gesture sampling.Source: §5.2.2 · p. 8
5.2.3 Ability of Multi-Dataset Training.
- Joint training with Trinity’s upper-body/audio pairs and AMASS’s body/hand data improves objective metrics on the BEATv1.3 test set.The datasets use separate VQ-VAEs and codebook-classification MLP heads.Source: §5.2.3 · p. 8
6. Conclusion
The conclusion presents EMAGE as a framework for completing partial gestures into audio-synchronized holistic motion. It also reports benefits from masked gesture reconstruction and multi-dataset training, alongside the release of BEAT2.
6. Conclusion
- EMAGE accepts partial gestures and completes them as audio-synchronized holistic gestures.The conclusion says masked gesture reconstruction can significantly enhance audio-conditioned generation performance.Source: §6 · p. 8
- Training EMAGE on multiple datasets further improves performance.The conclusion presents this as a capability of the framework.Source: §6 · p. 8
- BEAT2 is released as a multimodal gesture dataset consistent with SMPL-X and FLAME.The authors describe it as the largest such dataset and hope it supports knowledge and model sharing across subareas.Source: §6 · p. 8
Supplementary Material
The supplement contains seven sections covering evaluation, dataset details, reproduction, model settings, visualization, training time, and lower-body motion.
Supplementary Material
- Sections A–C cover evaluation metrics, BEAT2 dataset details, and baseline reproduction details.These sections provide supplementary material on measurement, data, and comparison methods.Source: Supplementary Material · p. 12
- Sections D–F cover EMAGE settings, a Blender visualization addon, and training time.The supplement lists these as separate sections.Source: Supplementary Material · p. 12
- Section G addresses the importance of lower-body motion.It is the final listed section of the supplement.Source: Supplementary Material · p. 12
A. Evaluation Metrics
The supplement defines metrics for gesture realism, diversity, and audio-motion synchrony, alongside facial-vertex errors. FGD compares distributions of latent features from real and generated gestures.
A. Evaluation Metrics
- FGD measures the distance between real and generated gesture-feature distributions using their means and covariances.The features come from a pretrained network; the real and generated distributions have separately defined first and second moments.Source: A. Evaluation Metrics · p. 12
- Higher L1 Diversity indicates greater variance across gesture clips, and its joint-position calculation uses zero translation to focus on local motion.Diversity is calculated over the test dataset.Source: A. Evaluation Metrics · p. 12
- Higher Beat Constancy indicates closer alignment between gesture rhythm and the audio beat.The method treats speech beginnings as audio beats and local minima in upper-body joint velocity, excluding fingers, as motion beats.Source: A. Evaluation Metrics · p. 12
B. BEAT2 Dataset Details
BEAT2 retains 60 hours from 25 speakers in standard and additional subsets, while its fitting procedures combine marker accuracy with motion plausibility and align FLAME expressions to ARKit.
Statistics
- 60 hours from 25 speakers remain after excluding five speakers for finger-data noise; BEAT2-standard contains 27 hours and BEAT2-additional 33 hours.The reported experiments use BEAT2-standard speaker 2; the dataset has 1,762 sequences averaging 65.66 seconds.Source: B. BEAT2 Dataset Details · p. 12
- The TalkShow comparison reports local and global diversity plus beat constancy; local and global diversity measure joint-position variance with and without global translations.The comparison is identified in Table 8.Source: Table 8 · p. 12, B. BEAT2 Dataset Details · p. 12
MoSh++ fitting
- MoSh++ balances marker fit, surface-distance energy, marker initialization, pose and shape priors, and velocity constancy in a weighted objective.The terms are used to balance accuracy and plausibility.Source: B. BEAT2 Dataset Details · pp. 12–13
FLAME parameter optimization
- FLAME expression parameters are optimized against equivalent vertices of an ARKit-compatible model, while fitted SMPL-X identity parameters are preserved.The expression optimization minimizes L2 distance between equivalent vertices.Source: B. BEAT2 Dataset Details · p. 13
C. Baselines Reproduction Details
The section describes baseline reproduction choices and EMAGE settings, including the representation-dependent comparison of training methods and the visualization add-on used for BEAT2.
C. Baselines Reproduction Details
- Baseline methods output 55-joint full-body rotations and global translations; subjective comparisons cover both upper-body-only and full-body results.Adversarial training is omitted for several baselines because it produced noticeable jitter despite increased velocity-loss weight.Source: C. Baselines Reproduction Details · p. 13
- With FLAME parameters, non-autoregressive training performs better; training behavior differs between VOCA vertex-loss and BEAT2 FLAME-plus-vertex-loss settings.The table reports vertex errors and states that non-autoregressive training demonstrates superior performance in BEAT2.Source: Table 9 · p. 13, C. Baselines Reproduction Details · p. 13
D. Settings of EMAGE
- EMAGE training runs for 400 epochs while the masked-joint ratio rises linearly from 0% to 95%; this outperforms a fixed 25% ratio in their experiments.The learning rate is 2.5e-4, using Adam with gradient norm clipping at 0.99.Source: D. Settings of EMAGE · pp. 13–14
- Four body segments use the same CNN-based VQ-VAE, while the global-motion predictor maps local motion and foot-contact labels to global translations.The VQ-VAE uses a 512-dimensional feature length and a codebook size of 256.Source: D. Settings of EMAGE · p. 14
E. Visualization Blender Add-on
- The Blender add-on visualization expands the SMPL-X model with 300 expression meshes as individual blendshape targets.The add-on’s latest version does not support the full range of SMPL-X facial expressions.Source: E. Visualization Blender Add-on · p. 14
F. Training time
The section reports EMAGE and VQ-VAE training times across hardware and speaker counts, then explains how lower-body motion relates to audio content and the other gesture components.
F. Training time
- 8.0h on a 4090 trains 1-speaker EMAGE for 400 epochs; 27.1h trains 25-speaker EMAGE for 100 epochs, with 20.1G memory and batch size 64.The table also reports training times for L4 and V100 GPUs.Source: F. Training time · p. 14
- 22.4 hours on five 4090 GPUs are required to pretrain the five VQ-VAEs for face, hands, upper body, lower body, and global motion.A separate table reports VQ-VAE training times across hardware and speaker counts.Source: F. Training time · p. 14
G. Importance of lower body motion
- Lower-body motion can align gestures with audio content, such as walking for hiking or kicking for football, although it is less strongly related to audio than upper-body motion.The section describes these aligned gestures as more vivid and impressive.Source: G. Importance of lower body motion · p. 14
- The lower-body decoder cross-attends to audio, upper-body, and hand latents; decoding directly from audio increases diversity but reduces coherence on BEATv1.3.Source: G. Importance of lower body motion · p. 14