PaperlayerView exact revision on arXiv

Source-linked AI summary

Generating Holistic 3D Human Motion from Speech

Hongwei Yi, Hualin Liang, Yifei Liu, Qiong Cao, Yandong Wen, Timo Bolkart, Dacheng Tao, Michael J. Black

arXiv:2212.04420v2cs.CVcs.GR

TL;DR

The paper addresses speech-driven generation of realistic and diverse holistic 3D body motion, including body, hands, and facial expressions. It contributes synchronized in-the-wild data and a separate face/body/hand generation framework using encoder-decoder, compositional VQ-VAE, and cross-conditional autoregressive modeling. Experiments and user studies report state-of-the-art qualitative and quantitative performance.

  • Problem

    Generating realistic and diverse holistic 3D body motions from speech is difficult because synchronized data are scarce and body parts correlate differently with speech.

  • Method

    TalkSHOW builds synchronized in-the-wild holistic mesh data and separately models face, body, and hand motions with encoder-decoder, compositional VQ-VAE, and cross-conditional autoregression.

  • Results

    The approach achieves state-of-the-art performance qualitatively and quantitatively, with evaluations covering realism and diversity against ground truth, baselines, and ablations.

  • Takeaways & Limitations

    TalkSHOW provides a holistic speech-to-motion framework for synthesizing realistic facial expressions, diverse body and hand motions, and coherent full-body animations.

  • Takeaways & Limitations

    The reconstruction and gesture representation assume consistent speaker identity and temporal smoothness, while MFCC features are used for body and hand generation.

Abstract

from arXiv · show

This work addresses the problem of generating 3D holistic body motions from human speech. Given a speech recording, we synthesize sequences of 3D body poses, hand gestures, and facial expressions that are realistic and diverse. To achieve this, we first build a high-quality dataset of 3D holistic body meshes with synchronous speech. We then define a novel speech-to-motion generation framework in which the face, body, and hands are modeled separately. The separated modeling stems from the fact that face articulation strongly correlates with human speech, while body poses and hand gestures are less correlated. Specifically, we employ an autoencoder for face motions, and a compositional vector-quantized variational autoencoder (VQ-VAE) for the body and hand motions. The compositional VQ-VAE is key to generating diverse results. Additionally, we propose a cross-conditional autoregressive model that generates body poses and hand gestures, leading to coherent and realistic motions. Extensive experiments and user studies demonstrate that our proposed approach achieves state-of-the-art performance both qualitatively and quantitatively. Our novel dataset and code will be released for research purposes at https://talkshow.is.tue.mpg.de.

1. Introduction

TalkSHOW targets realistic and diverse holistic 3D motion from speech by addressing scarce synchronized data and differing speech correlations across body parts. It separates face, body, and hand modeling and reports state-of-the-art qualitative and quantitative performance.

  • Holistic speech-to-motion generation must coordinate facial expressions, body posture, and hand movement for realistic human communication.The paper emphasizes that full 3D surface motion synchronized with speech is important for realistic virtual agents.
  • The task is challenging because synchronized holistic datasets are scarce, human shapes and deformable parts vary, and body-part correlations with speech differ.These challenges jointly complicate realistic and stable holistic mesh generation.
  • The authors introduce in-the-wild 3D holistic body mesh annotations with synchronous audio to support data-driven speech-to-motion modeling.The dataset also supports broader research such as realistic digital human rendering.
  • TalkSHOW separately models face, body, and hands because facial motion is strongly speech-correlated whereas body and hand motion are less correlated.This design matches the different cross-modal relationships among body parts.
  • A face encoder-decoder uses pretrained wav2vec 2.0, while compositional VQ-VAE and cross-conditional autoregression generate diverse and coherent body-hand motion.The discrete representation supports multiple motion modes, which are sampled through the autoregressive model.
  • Extensive quantitative, qualitative, and user-study evaluations demonstrate state-of-the-art quality for speech-synthesized full expressive 3D character animations.The evaluation covers realism and diversity against ground truth, baselines, and ablations.

2. Related work

Related work spans holistic mesh recovery, speech-to-motion datasets, and body-motion generation. TalkSHOW distinguishes itself with holistic coverage and separate, part-specific generation using compositional and cross-conditional models.

  • SMPLify-X fits SMPL-X to detected 2D keypoints, while PIXIE and PyMAF-X use regression or alignment mechanisms for holistic recovery.TalkSHOW adapts optimization-based SMPLify-X to videos of talking persons.
  • In-house speech-to-motion datasets offer accurate annotations but limited scale, while many datasets cover only heads or bodies rather than the whole body.Their multi-camera capture systems are expensive and labor intensive.
  • TalkSHOW’s dataset combines holistic body meshes and synchronized speech from in-the-wild videos, covering body poses, hand gestures, and facial expressions.This contrasts with prior representations that limit possible applications of generated talking motions.
  • Existing body-motion methods include rule-based mappings to motion units and learning-based approaches, with rule-based systems offering explainability and controllability.The related-work discussion contrasts these approaches with holistic generation.
  • TalkSHOW models body parts according to their speech correlations, using a deterministic face encoder-decoder and two VQ-VAEs for diverse body and hand motions.The face is treated as more speech-correlated than the body parts.
  • Its compositional quantized space is more expressive for conversational gestures, while cross-conditional autoregression produces more fluid and natural body-part motions.The autoregressive model is presented as a distinction from previous VQ-VAE-based methods.

3. Dataset

The dataset provides expressive holistic meshes with synchronized audio reconstructed from in-the-wild talking videos. SHOW improves reconstruction through initialization, richer data terms, and video-based regularization, yielding improved pseudo ground truth.

  • The dataset contains expressive 3D body meshes at 30fps paired with synchronized audio sampled at 22KHz from in-the-wild monocular videos.These meshes serve as pseudo ground truth for speech-to-motion generation.
  • Filtering videos for resolution, hand occlusion, and valid links produces a 26.9-hour dataset from four speakers.Raw videos are cropped into clips shorter than ten seconds for mini-batch processing.
  • Holistic pseudo ground truth represents connected face, hand, and body meshes using SMPL-X parameters.The representation includes shared body shape and frame-wise motion parameters.
  • SHOW adapts SMPLify-X to talking-person videos to improve the stability, accuracy, and efficiency of whole-body reconstruction.Advanced regression predictions initialize the SMPLify-X optimization.
  • The reconstruction data term combines joint reprojection with body silhouettes, facial landmarks, facial shapes, and photometric face loss.These additional cues target body and facial detail capture.
  • Video regularization assumes consistent speaker identity and temporal smoothness of body pose, facial expression, and lighting.These assumptions are used to derive more reasonable regularizations.
  • The reported practices significantly improve the reconstructed pseudo ground truth.The improvement is summarized with reference to Figure 9.

4. Method

TalkSHOW separates face generation from body-and-hand generation, using audio and speaker identity to synthesize holistic motion. Body and hand motions are represented with compositional VQ-VAE codebooks and generated through cross-conditional autoregressive prediction.

  • Overview: TalkSHOW models facial motion with an encoder-decoder and body-and-hand motion with a separate generator.The separation reflects the stronger speech correlation of facial articulation compared with body and hand motion.
  • Face Generator: The face generator uses wav2vec 2.0 speech features, speaker identity, temporal convolutional networks, and an MSE loss.The speech encoder produces a 768-dimensional representation projected to 256 dimensions before decoding.
  • Cross-Conditional Autoregressive Modeling: At inference, sampled body and hand code indices are decoded to produce diverse future motions.The model uses Gated PixelCNN with masked convolutions so future information is unavailable during prediction.
  • Body and Hand Generator: Separate VQ-VAEs encode body and hand motions into compositional codebooks whose combinations expand the range of motion diversity.The two codebooks provide |Zb| × |Zh| possible body-hand pose code pairs.
  • Body and Hand Generator: The body-and-hand generator quantizes embeddings from temporal windows and decodes the resulting codebook features into synthesized motion.The method uses w = 4, with each embedding representing four consecutive poses.
  • Cross-Conditional Autoregressive Modeling: A cross-conditional autoregressive model predicts body and hand code indices from audio, identity, and preceding motion codes.Each modality conditions on the other to preserve synchronization and coherence between body and hand motions.

5. Experiments

Experiments evaluate TalkSHOW’s realism, synchronization, and diversity using quantitative metrics, qualitative examples, ablations, and perceptual studies. The method outperforms baselines and produces diverse, speech-consistent holistic motions.

  • Evaluation setup: TalkSHOW evaluates facial realism and synchronization alongside body-and-hand realism and diversity.Metrics include L2 and LVD for facial motion, and RS and Variation for body and hand motion.
  • Quantitative comparison: TalkSHOW outperforms Habibie et al. across all reported metrics, including facial accuracy, motion diversity, and body-hand realism.The authors attribute the gains to the face generator, compositional motion representation, and cross-conditional autoregressive model.
  • Quantitative comparison: TalkSHOW achieves larger realism and diversity gains than VAE-based models, especially in diversity.The compositional codebooks memorize multiple body and hand motion modes.
  • Qualitative analysis: Generated motions respond to speech rhythm while varying in motion range and hand usage across samples.For the strengthening tone of “But,” samples raise and lower the hands and may use either hand or both hands.
  • Qualitative analysis: Generated facial motions show consistent lip movements for corresponding phonemes and generalize to French and songs.The face generator is illustrated with phonemes including /f/, /t/, /b/, and /æ/.
  • Ablation studies: The wav2vec-based face model improves L2 from 0.165 to 0.130 and LVD from 0.277 to 0.251 over the MFCC-based model.The comparison uses an additional encoder for MFCC features to increase their dimension from 64 to 256.

6. Conclusion

The conclusion presents TalkSHOW as a speech-to-holistic-mesh system combining accurate facial motion with diverse and coherent body-hand motion. It also contributes synchronized in-the-wild mesh data and reports state-of-the-art qualitative and quantitative performance.

  • Conclusion: TalkSHOW generates 3D holistic body meshes from speech, including face, hands, and body.The paper describes it as the first approach with this capability.
  • Conclusion: An encoder-decoder produces accurate lip shapes, while compositional VQ-VAE and cross-conditional modeling support diverse and coherent body-hand motion.The components are assigned to facial generation, diversity, and coherent prediction respectively.
  • Conclusion: The work contributes accurate 3D holistic body meshes with synchronized audio from in-the-wild videos.The annotations use an empirical approach designed for videos.
  • Conclusion: Experimental results demonstrate state-of-the-art performance qualitatively and quantitatively.

A.1. Dataset Description

The dataset comprises in-the-wild talking videos reconstructed as SMPL-X holistic meshes. The reconstruction pipeline combines multiple initial estimators with silhouette, facial, photometric, temporal-smoothness, and collision objectives.

  • Dataset: The dataset contains 26.9 hours of high-quality 3D holistic mesh annotations from four persons, with clips shorter than 10 seconds.Figure 7 reports clip-duration distributions for different characters.
  • Representation: SMPL-X represents the face, hands, and body with 10,475 vertices and 67 joints.Mesh vertices are produced by linear blend skinning with learned corrective blend shapes.
  • Representation: Video sequences are represented temporally through mesh vertices, joints, body poses, and hand poses, with fixed invisible lower-body poses.
  • Optimization: Initialization combines PIXIE and PyMAF-X for body and hand pose with DECA for jaw pose and facial expression.These estimates initialize the optimization-based reconstruction.
  • Optimization: The objective combines single-image priors with facial, silhouette, photometric, temporal-smoothness, and hand-collision losses.The weights λ control each term’s influence.
  • Optimization: Temporal smoothness terms reduce jitter from noisy 2D keypoints, while collision penalization prevents inter-penetration between the hands.

B. Network Architecture Details

The network uses separate facial and body-hand generation components. Audio features drive a face encoder-decoder, while compositional VQ-VAEs and a masked autoregressive model generate body and hand motions.

  • Face generator: The face generator normalizes raw audio, extracts features, interpolates them to target frames, and decodes motion with temporal convolutions.The decoder uses six temporal convolution layers.
  • Reconstruction comparison: The reconstruction system produces more accurate and stable holistic results with details than SMPLify-X, PIXIE, and PyMAF-X.The comparison is shown for four subjects.
  • Face generator: The face generator is trained for 100 epochs with batch size 1 using full-length audio and corresponding facial motions.
  • Body and hand generator: Separate VQ-VAEs encode body and hand motions into compositional discrete representations.Each encoder uses residual and temporal convolution layers with batch normalization and Leaky ReLU activation.
  • Autoregressive model: The autoregressive model concatenates MFCC audio features with VQ-VAE encoder outputs and predicts codebooks using a Gated PixelCNN.Masked convolutions prevent access to future information.

C. More Comparison

The comparisons show that TalkSHOW produces more accurate, stable, realistic, diverse, and audio-aligned holistic motions than the evaluated baselines. Its generated meshes also support photo-realistic neural-avatar rendering.

  • Comparison with prior work: TalkSHOW produces more stable and accurate holistic body meshes than Habibie et al. [26], whose separate representations yield unnatural face/hand-body poses and jittery videos.The comparison specifically highlights improved hand and head poses reconstructed from the estimated 3D skeleton.
  • Quantitative evaluation: TalkSHOW outperforms the baselines on Frechet Gesture Distance and beat consistency while generating more diverse motions better aligned with the input audio.FGD measures motion realism, while beat consistency measures alignment between generated body motion and input audio.
  • Application: TalkSHOW meshes and colors can be passed to SMPLpix to render photo-realistic human images under different generated poses.The application projects colored mesh vertices onto the image plane before neural rendering.

E. Discussions

The discussion identifies limitations in reconstruction, camera settings, facial modeling, and potential misuse. These boundaries motivate future extensions and responsible deployment.

  • Reconstruction: The reconstruction is sensitive to severe hand-shape deformation and heavy occlusion because SHOW relies on SMPLify-X with 2D keypoint reprojection supervision.The authors suggest advanced hand models with richer shape and pose spaces as a future direction.
  • Camera setting: SHOW currently handles only static-camera cases, with moving-camera support left for future work.The stated scope boundary concerns the camera setting rather than the generated motion types.
  • Facial modeling: The face generator focuses mainly on facial motion such as lip motion and may not handle complex emotion-driven facial movements.The authors plan to extend the model to these facial-motion components.
  • Risks and potential misuse: Realistic and diverse audio-driven motion could be misused to create highly realistic fake videos depicting events that never happened.The authors recommend using the technology responsibly and carefully.
Loading 2212.04420v2…