Source-linked AI summary
MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement
Alexander Richard, Michael Zollhoefer, Yandong Wen, Fernando de la Torre, Yaser Sheikh
TL;DR
Existing speech-driven facial animation methods face limited upper-face realism, underdetermined facial motion, or dependence on personalized capture and models. MeshTalk uses a categorical latent space and cross-modality loss to disentangle audio-correlated from audio-uncorrelated motion. It achieves highly accurate lip motion and plausible whole-face animation, outperforming baselines with more than 75% preference in key comparisons.
Problem
Existing audio-driven facial animation methods can produce uncanny or static upper-face motion, struggle with one-to-many facial outcomes, or require person-specific models and capture data.
Method
MeshTalk learns a categorical facial-animation latent space with cross-modality and eyelid losses, then autoregressively samples audio-conditioned motion over that space.
Results
More than 75% of user-study cases ranked MeshTalk better than or equal to VOCA for full-face animation and lip sync, with 66.4% for upper-face motion.
Takeaways & Limitations
The method generates highly accurate lip motion together with plausible motion in audio-uncorrelated facial regions and achieves state-of-the-art quality.
Takeaways & Limitations
A required 100ms audio look-ahead introduces latency and prevents online use, while low-cost commodity hardware cannot run the approach in real time.
Abstract
from arXiv · showhide
This paper presents a generic method for generating full facial 3D animation from speech. Existing approaches to audio-driven facial animation exhibit uncanny or static upper face animation, fail to produce accurate and plausible co-articulation or rely on person-specific models that limit their scalability. To improve upon existing models, we propose a generic audio-driven facial animation approach that achieves highly realistic motion synthesis results for the entire face. At the core of our approach is a categorical latent space for facial animation that disentangles audio-correlated and audio-uncorrelated information based on a novel cross-modality loss. Our approach ensures highly accurate lip motion, while also synthesizing plausible animation of the parts of the face that are uncorrelated to the audio signal, such as eye blinks and eye brow motion. We demonstrate that our approach outperforms several baselines and obtains state-of-the-art quality both qualitatively and quantitatively. A perceptual user study demonstrates that our approach is deemed more realistic than the current state-of-the-art in over 75% of cases. We recommend watching the supplemental video before reading the paper: https://github.com/facebookresearch/meshtalk
1. Introduction
Speech-driven facial animation must produce natural, intelligible motion because viewers are sensitive to subtle facial cues and disturbing artifacts. MeshTalk addresses limitations of person-specific or over-smoothed approaches by disentangling audio-correlated and uncorrelated facial motion.
- Speech-driven facial animation supports applications ranging from games and e-commerce to immersive VR telepresence, with demanding settings requiring naturalness, plausibility, and intelligibility.
- Poor co-articulation or lip-sync is disturbing because the human visual system is adapted to subtle facial motions and expressions.
- Person-specific methods can produce high-quality lip and upper-face animation but require extensive paired audio–mesh capture, limiting practicality for consumer-facing applications.
- Generic methods can animate arbitrary users from speech and a neutral scan, but often produce uncanny or static upper-face motion because speech leaves facial expressions underdetermined.
- MeshTalk learns a categorical latent space with cross-modality loss to separate audio-correlated mouth motion from audio-uncorrelated movements such as blinks and eyebrow motion.
2. Related Work
Prior speech-driven facial animation spans stylized viseme systems, 2D neural talking heads, and 3D avatar methods. These approaches trade off generality, 3D compatibility, personalization, fidelity, and upper-face realism, motivating MeshTalk’s non-personalized design.
- Viseme-based systems generate speech-linked facial motion from text or speech and synthesize it by blending templates or context-dependent viseme models.
- Neural 2D talking-head methods address lip synchronization and facial video synthesis in pixel space but do not easily generalize to 3D.
- Recent 3D avatar methods can achieve realistic results using visual sensors, but are typically personalized and not audio-driven.
- Fully speech-driven 3D techniques commonly require personalized models, lower-fidelity blendshapes or landmarks, or large datasets with hundreds of training subjects.
- MeshTalk differs by being non-personalized while generating realistic upper-face motion and highly accurate lip synchronization.
3. Method
MeshTalk maps speech and facial expression signals into a multi-head categorical latent space, then decodes sampled codes onto an arbitrary neutral template mesh. Cross-modality training separates audio-dependent mouth motion from audio-independent upper-face motion, while autoregressive sampling supplies plausible missing expressions at inference.
- Categorical expression space: The latent space uses H categorical heads with C labels each, yielding C^H configurations; the paper uses C = 128 and H = 64.Multiple heads provide a large expression space without requiring one infeasibly large categorical layer.
- Model overview: The model encodes speech and animated face meshes into a categorical latent representation and decodes expression codes onto a provided neutral-face template.The decoder generates a mesh sequence matching the template identity while following the encoded expression code.
- Cross-modality loss: Cross-modality training reconstructs the upper face independently of audio and the mouth area from audio independently of the expression sequence.Randomly mismatched expression and audio inputs create separate reconstruction targets, with masks emphasizing upper-face and mouth vertices.
- Cross-modality loss: An additional eyelid loss emphasizes sparse, fast eye-blink events, and the final objective combines cross-modality and eyelid losses with equal weighting.The eyelid mask assigns weight to eyelid vertices and zero weight elsewhere.
- Audio-conditioned autoregressive modeling: At inference, an audio-conditioned autoregressive temporal model samples categorical expression codes so the decoder can synthesize missing non-audio-correlated motion.The model uses current and past audio context and previously generated categorical labels, rather than future audio context.
4. Evaluation
The evaluation uses a large, high-fidelity multi-subject dataset to test latent-space design, disentanglement, lip synchronization, perceptual quality, and retargeting. Results show that cross-modality training and categorical latents support structured facial control, while the approach produces realistic full-face motion and competitive lip synchronization.
- Evaluation setup: The dataset contains 250 subjects, 13 hours of paired audio-visual data, and 1.4 million frames captured from 80 synchronized cameras.The 6,172-vertex face model includes eyelids, upper-face structure, and varied hairstyles.
- Latent-space evaluation: The multi-modal encoder and cross-modality loss prevent the latent space from entangling eye motion with mouth shape and improve audio-conditioned generation.Without audio encoder input or with a simple reconstruction loss, the model can ignore audio and produce less accurate or muted facial motion.
- Latent-space evaluation: Audio mainly controls the mouth while expression controls the upper face, with some boundary representations affecting both regions.The latent configurations associated with audio and expression changes form separated clusters with minimal leakage.
- Latent-space evaluation: The categorical latent space yields lower lip and overall vertex errors than the continuous alternative.The continuous model also produces less varied upper-face motion in audio-driven meshes.
- Audio-driven evaluation: The approach achieves lower average per-frame lip error than VOCA on highly detailed meshes, where VOCA often misses lip closures.Lip error is measured using the maximum ℓ2 error among lip vertices per frame because average vertex errors can mask inaccurate lip shapes.
- Audio-driven evaluation: Participants ranked the approach better than or equal to VOCA in 77% of cases, including full-face animation and lip synchronization.For upper-face motion alone, it was ranked better or equal in 66.4% of cases; qualitative examples show diverse blinks and eyebrow raises.
5. Limitations
The approach has three stated limitations: 100ms inference latency, insufficient real-time performance on low-cost hardware, and dependence on successful tracking of facial regions.
- 100ms of audio look-ahead creates inherent latency and prevents online applications.The look-ahead improves lip-sync quality, including modeling lip closure for sounds such as /p/.
- The method cannot run in real time on low-cost commodity hardware such as laptop CPUs or virtual reality devices.The authors suggest further research could substantially reduce computational cost.
- Tracking failures caused by occlusions, such as hair covering eyebrows or eyes, prevent correct learning of motion correlations with audio.
6. Conclusion
The paper presents a generic audio-to-3D-face animation method using a categorical latent space and cross-modality loss. It produces accurate lip motion and plausible motion in audio-uncorrelated facial regions, outperforming several baselines with state-of-the-art quality.
- The method generates 3D facial animation from audio alone using a categorical latent space and cross-modality loss.The approach uses autoregressive generation to produce realistic animation.
- It achieves highly accurate lip motion while synthesizing plausible motion in facial regions uncorrelated with audio.
- The approach outperforms several baselines and obtains state-of-the-art quality.
- The authors position the method as a step toward VR telepresence as head-mounted capture devices become smaller.Smaller devices complicate accurate lip and tongue regression from oblique camera viewpoints.