Source-linked AI summary
Capture, Learning, and Synthesis of 3D Speaking Styles
Daniel Cudeiro, Timo Bolkart, Cassidy Laidlaw, Anurag Ranjan, Michael J. Black
TL;DR
Realistic audio-driven 3D facial animation lacks general datasets and models that work across speakers, languages, and face shapes. The paper introduces VOCASET and trains VOCA by separating identity from facial motion, achieving realistic animation with style and pose controls while retaining limitations in upper-face and inferred head motion.
Problem
Realistic speech-driven 3D facial animation remains unsolved because general methods must handle nonlinear speech-motion mappings, many-to-many phoneme correspondences, multiple people and styles, and realistic faces.
Method
VOCA trains an encoder-decoder network on multi-subject 4D scans, factors identity from facial motion, conditions on subject labels, and uses DeepSpeech and FLAME for audio processing and face representation.
Results
VOCA produces realistic facial motion for unseen subjects, with recorded performances preferred at 83 ± 9% versus predictions at 17 ± 9%, while DeepSpeech features were preferred at 78 ± 16% versus fbank at 22±16%.
Takeaways & Limitations
VOCA generalizes across speech sources and languages and supports animator control of speaking style, identity-dependent shape, expression, and pose for adult faces.
Takeaways & Limitations
VOCA does not model upper-face motions well because emotion is absent from the controlled capture data, and it does not infer head motion from audio.
Abstract
from arXiv · showhide
Audio-driven 3D facial animation has been widely explored, but achieving realistic, human-like performance is still unsolved. This is due to the lack of available 3D datasets, models, and standard evaluation metrics. To address this, we introduce a unique 4D face dataset with about 29 minutes of 4D scans captured at 60 fps and synchronized audio from 12 speakers. We then train a neural network on our dataset that factors identity from facial motion. The learned model, VOCA (Voice Operated Character Animation) takes any speech signal as input - even speech in languages other than English - and realistically animates a wide range of adult faces. Conditioning on subject labels during training allows the model to learn a variety of realistic speaking styles. VOCA also provides animator controls to alter speaking style, identity-dependent facial shape, and pose (i.e. head, jaw, and eyeball rotations) during animation. To our knowledge, VOCA is the only realistic 3D facial animation model that is readily applicable to unseen subjects without retargeting. This makes VOCA suitable for tasks like in-game video, virtual reality avatars, or any scenario in which the speaker, speech, or language is not known in advance. We make the dataset and model available for research purposes at http://voca.is.tue.mpg.de.
1. Introduction
VOCA addresses the missing general method for mapping speech from arbitrary speakers and languages to realistic 3D facial motion across face shapes. It introduces VOCASET and a model that factors identity from motion while supporting speaker-style variation and facial controls.
- The paper targets a general method linking speech from any person and language to 3D facial motion of arbitrary face shapes.
- VOCASET contains 4D face scans and synchronized speech from 12 subjects across 480 short sequences captured at 60 fps.Sentences were selected using protocols designed to maximize phonetic diversity, and scans share a common template correspondence.
- VOCA trains a deep neural network that generalizes facial animation to new speakers and avoids prior methods’ subject-specific or mouth-only limitations.
- Factoring facial identity from motion lets VOCA learn speech-to-motion mappings without conflating subject appearance and movement.
- Subject-label conditioning combines multiple speakers during training, enabling synthesis for unseen subjects and different speaking styles.
2. Related work
Prior facial-animation work uses video, text, audio, or performance capture, often with subject-specific rigs, limited facial regions, or restricted generalization. VOCA instead uses audio to animate realistic 3D avatars across subjects.
- Existing approaches span speech-, text-, video-, and performance-based facial animation, with many relying on visual data or personalized rigs.
- Video-based methods include phoneme-driven mouth animation, audio-to-motion regression, and photorealistic mouth textures, but some do not generalize to unseen faces or viewpoints.
- Some audio-driven 3D methods use multi-view capture or high-resolution scans, but subject-specific models require extensive subject data for new faces.
- Text-driven methods animate faces from written input, whereas VOCA’s primary focus is speech-driven animation of realistic face meshes.
- Compared with visual retargeting methods, VOCA relies solely on audio to animate digital avatars.
- Public 3D datasets commonly emphasize static shape or emotional dynamics, while VOCASET emphasizes speech-related facial motion and high-quality 4D capture.4DFAB has 180 subjects but only nine word utterances per subject; B3D(AC)^2 contains 40 spoken English sentences and raw-scan artifacts.
3. Preliminaries
VOCA seeks subject-independent generalization across speaker audio variation and facial shape or motion. Its preliminaries use DeepSpeech for robust audio features and FLAME to represent variable facial identity and expression.
- The goal is to generalize to unseen subjects across accent, speed, audio source, noise, environment, facial shape, and facial motion.
- DeepSpeech is integrated to improve robustness to different audio sources, recording artifacts, noise, and language.It is an end-to-end automatic speech recognition model whose output is a character-probability distribution.
- FLAME provides a common learning space for variation in facial shape, expression, motion, and speaking style across subjects.
4. VOCA
VOCA converts DeepSpeech features and a subject-specific template into 3D vertex displacements through an encoder-decoder network. Subject conditioning and FLAME integration enable style, identity shape, expression, and pose controls.
- Overview: VOCA takes a subject-specific template and raw audio, extracts DeepSpeech features, and predicts a target 3D mesh.
- Encoder: The encoder uses convolutional and fully connected layers to map speech features into a low-dimensional embedding.Subject labels condition the network so it can learn subject-specific speaking styles across multiple training subjects.
- Speech feature extraction: Speech features are resampled from 50 to 60 frames per second and organized into overlapping temporal windows.For an audio clip of T seconds, the resulting array has dimensions 60T × W × D.
- Decoder: The decoder maps the embedding to a 5023 × 3 array of vertex displacements from the input template.Its weights are initialized with 50 PCA components computed from training vertex displacements.
- Animation control: During inference, changing the subject code alters speaking style, while FLAME enables edits to identity-dependent shape, expression, and head, jaw, and eyeball pose.
5. Model training
VOCA is trained on audio–4D scan pairs with disjoint subject and sentence splits, using position and velocity losses to match facial motion and encourage temporal stability.
- Training uses audio–4D scan pairs, with each audio window centered on a video frame and paired with 3D vertex positions.
- Eight subjects provide 320 training sentences, while validation and test sets each use 40 unique sentences from held-out subjects.Subjects and sentences are fully disjoint across training, validation, and testing.
- The position loss measures predicted-versus-training vertex distances, encouraging the model to match the ground-truth performance.
- The velocity loss compares consecutive-frame differences between predictions and training vertices, inducing temporal stability.
- VOCA is trained for 50 epochs with learning rate 1e −4, batch size 64, and a 16-frame window containing 29 speech features.
6. VOCASET
VOCASET is a 4D audio-scan dataset of diverse English speech from 12 subjects, captured at 60 fps and processed into aligned, unposed meshes that preserve subtle mouth motion.
- VOCASET contains 40 English speech sequences per subject from 6 female and 6 male speakers, with sentences selected for phonetic diversity.Each sequence lasts three to five seconds and combines TIMIT sentences, pangrams, and SQuAD questions.
- Sentences are distributed across subjects to combine shared speech content with 200 sentences spoken by only one or two subjects.
- The capture system records synchronized audio and 3D meshes at 60 fps using multi-camera active stereo.
- Raw head scans are registered with the generic FLAME model, producing meshes with 5023 3D vertices.
- Meshes are unposed into zero pose, with eyes smoothed for noise removal while the mouth remains unsmoothed to preserve subtle motions.
7. Experiments
Experiments evaluate VOCA perceptually and qualitatively, showing generalization across faces and languages, controllable speaking styles, noise robustness, and realistic animation under shape and pose changes. Comparisons also expose limitations: recorded subject-specific performances remain more natural, and style recognition varies across conditions.
- Perceptual evaluation: 400 HITs rated recorded performances more natural than VOCA predictions, 83 ± 9% versus 17 ± 9%, because predictions miss subtle subject-specific details.
- Perceptual evaluation: 400 HITs rated VOCA with DeepSpeech features more natural than the fbank variant, 78 ± 16% versus 22±16%, indicating better generalization to unseen audio.
- Perceptual evaluation: Style recognition varied by condition: Turkers consistently identified matching styles in some conditions, while choices were at chance in others, with content usually having no significant impact.
- Qualitative evaluation: VOCA generalizes across large face-shape variations in BU-3DFE templates, while the same audio drives animation frames for each selected subject.
- Qualitative evaluation: VOCA produces plausible animation for non-English sentences, demonstrating generalization across languages.
- Qualitative evaluation: Different subject conditions produce distinct lip-articulation styles, and convex combinations provide intermediate styles with varying mouth amplitudes.
- Qualitative evaluation: Only high street-noise levels damp facial motion; animations remain plausible despite noise.
- Qualitative evaluation: Changing identity shape or head pose while keeping audio fixed preserves realistic facial animation.
8. Discussion
VOCA produces realistic lower-face speech animation across adult identities, but conversational realism remains limited by weakly audio-correlated cues and missing emotion. Subject conditioning enables style control, while broader style coverage and richer conversational modeling remain open directions.
- VOCA lacks some conversational details because emotion is absent from the controlled capture data.Upper-face motions such as eye and eyebrow movements are not strongly correlated with audio, so the model primarily learns speech-driven lower-face motion.
- Head motion and other non-verbal cues are weakly correlated with audio, so VOCA does not infer them from speech.Animators can add head motion manually, while speech-independent head-motion models and richer interaction models are identified as future work.
- Subject-label conditioning lets users alter speaking style during inference.Increasing the number of subjects and learning to mitigate, amplify, or combine styles remain future tasks.
9. Conclusion
The paper presents VOCA, a generic speech-driven facial animation framework that separates identity from motion and works across identities. It generalizes across speech sources, languages, and 3D face templates while offering controls for style, shape, and pose.
- VOCA automatically converts arbitrary speech and a static character mesh into realistic animation across a range of identities.The framework is trained on the multi-subject VOCASET 4D face dataset.
- Factoring identity from facial motion enables animation of a wide range of adult faces.Conditioning on subject labels supports training across multiple subjects and synthesizing different speaker styles at test time.
- VOCA generalizes across speech sources, languages, and 3D face templates, with optional controls for style, shape, and head pose.The dataset, trained model, and code are available for research purposes.