Source-linked AI summary
FaceFormer: Speech-Driven 3D Facial Animation with Transformers
Yingruo Fan, Zhaojiang Lin, Jun Saito, Wenping Wang, Taku Komura
TL;DR
Speech-driven 3D facial animation needs longer audio context than short-window approaches provide, while 3D audio-visual data remain scarce. FaceFormer addresses both challenges with an autoregressive Transformer using pretrained speech representations and tailored attention, achieving higher lip-synchronization quality and more realistic animation than state-of-the-art methods.
Problem
Short audio windows and limited 3D audio-visual data constrain speech-driven 3D facial animation, despite the need for longer context to animate the whole face realistically.
Method
FaceFormer is an autoregressive Transformer that uses pretrained speech representations, long-term audio context, face-motion history, and biased attention with periodic positional encoding.
Results
FaceFormer demonstrates higher-quality lip synchronization and more realistic facial animation than state-of-the-art methods.
Takeaways & Limitations
Tailored attention and pretrained speech representations support realistic, temporally stable speech-driven animation across the whole face.
Takeaways & Limitations
The technology could potentially be misused to cause embarrassment, motivating attention to responsible use.
Abstract
from arXiv · showhide
Speech-driven 3D facial animation is challenging due to the complex geometry of human faces and the limited availability of 3D audio-visual data. Prior works typically focus on learning phoneme-level features of short audio windows with limited context, occasionally resulting in inaccurate lip movements. To tackle this limitation, we propose a Transformer-based autoregressive model, FaceFormer, which encodes the long-term audio context and autoregressively predicts a sequence of animated 3D face meshes. To cope with the data scarcity issue, we integrate the self-supervised pre-trained speech representations. Also, we devise two biased attention mechanisms well suited to this specific task, including the biased cross-modal multi-head (MH) attention and the biased causal MH self-attention with a periodic positional encoding strategy. The former effectively aligns the audio-motion modalities, whereas the latter offers abilities to generalize to longer audio sequences. Extensive experiments and a perceptual user study show that our approach outperforms the existing state-of-the-arts. The code will be made available.
1. Introduction
FaceFormer addresses the challenges of long-term audio context and scarce 3D audio-visual data in speech-driven 3D facial animation. Its autoregressive Transformer combines pretrained speech representations and biased attention mechanisms to generate realistic, temporally stable whole-face motions with accurate lip movements.
- Motivation: Existing 2D talking-head methods are not directly applicable to 3D games and VR, while monocular-video reconstruction can produce unreliable 3D facial parameters.Synthetic 3D quality is limited by reconstruction techniques that may miss subtle 3D changes.
- Method: FaceFormer autoregressively predicts 3D face meshes by encoding long-term audio context and face-motion history.The architecture targets realistic, temporally stable animation of both upper- and lower-face expressions.
- Method: Self-supervised pretrained speech representations address limited 3D audio-visual data and improve difficult mouth movements, including fully closed lips for /b/, /m/, and /p/.The motivation notes that vanilla Transformers are data-hungry for this task.
- Method: Biased cross-modal attention aligns audio and motion, while biased causal self-attention with periodic positional encoding improves generalization to longer audio sequences.These mechanisms are designed specifically for speech-driven 3D facial animation.
- Evaluation: Experiments and a perceptual user study report superior realistic facial animation and lip sync over existing state-of-the-art methods on two 3D datasets.The evaluation assesses synthesized face-motion quality.
2. Related Work
Prior 3D facial-animation work includes controllable procedural methods and data-driven models, but existing approaches face manual-tuning, limited-context, or data-demand challenges. Transformers offer long-range sequence modeling relevant to aligning speech with facial motion.
- 3D Facial Animation: Procedural methods animate talking mouths with explicit rules and speech-control parameters, providing direct control over mouth-movement accuracy.Their explicit control is an appealing strength, but parameter tuning requires substantial manual effort.
- 3D Facial Animation: Data-driven approaches generate 3D facial animation from audio and related features, including speaker-independent models that capture varied speaking styles.VOCA captures a variety of speaking styles, although its generated motions are mostly concentrated in the lower face.
- 3D Facial Animation: The most closely related high-resolution methods represent outputs as high-dimensional 3D vertex vectors, but some use short audio windows while another requires large amounts of high-fidelity facial data.The short-window models ignore long-term audio context, whereas the data-intensive model uses substantial 3D facial data to support quality and generalization.
- Transformers: Transformers model long-range contextual information by attending to all input tokens in parallel, unlike recursively processed RNN sequences.Transformer-based methods have also been explored for 3D body-motion synthesis and 3D dance generation.
- Transformers: Speech-driven 3D facial animation additionally requires audio–face-motion alignment for accurate lip movements while retaining long-term audio context.This alignment requirement distinguishes the task from dance generation, whose output motion is highly unconstrained.
3. Our Approach: FaceFormer
FaceFormer formulates speech-driven 3D facial animation as autoregressive seq2seq prediction, generating facial movements from audio context, speaker style, and prior motions. Its Transformer architecture combines wav2vec 2.0 speech encoding with periodic temporal and cross-modal attention biases to align modalities and support longer sequences.
- Architecture: FaceFormer autoregressively predicts 3D facial movements from encoded audio, speaker identity, and previously generated facial movements.The encoder-decoder model transforms raw audio into speech representations, while a style embedding represents speaker identities.
- Audio Encoder: The audio encoder follows wav2vec 2.0, combining temporal convolutions with a multi-layer Transformer encoder for speech representation learning.A randomly initialized linear projection is added above the pretrained encoder, and interpolation resamples audio features when audio and motion frequencies differ.
- Biased Attention: Biased causal self-attention assigns negative-infinity weights to future frames and periodic static biases to lower-triangular attention, emphasizing nearby motion periods.The proposed temporal bias generalizes ALiBi, which becomes a special case when p = 1.
- Biased Attention: Biased cross-modal multi-head attention aligns audio and motion by adding an alignment bias to query-key attention scores.Audio tokens encode long-term speech context, motion tokens encode facial-motion history, and the resulting hidden state is projected into the 3D vertex space.
- Training: FaceFormer is trained autoregressively by minimizing Mean Squared Error between predicted and ground-truth 3D facial motion sequences.The authors report that the less guided autoregressive scheme performs better than fully guided teacher forcing.
4. Experiments and Results
Experiments on BIWI and VOCASET evaluate FaceFormer against VOCA and MeshTalk using lip-sync metrics, user studies, and qualitative comparisons. Ablations indicate that long-range attention, wav2vec initialization, alignment bias, autoregressive decoding, and periodic positional encoding contribute to synchronized, stable, and expressive facial motion.
- Datasets: Experiments use BIWI and VOCASET, two public datasets pairing English speech utterances with 3D facial scans.BIWI contains 40 sentences and fewer phonemes, making it more challenging for lip synchronization; VOCASET contains 255 sentences.
- Baselines: FaceFormer is compared with VOCA and MeshTalk on both datasets, with FaceFormer and VOCA conditioned on training speaker identities during inference.For unseen subjects, predictions from FaceFormer and VOCA are conditioned on all training identities.
- Quantitative evaluation: FaceFormer achieves lower average lip error than VOCA and MeshTalk on BIWI-Test-A, indicating more accurate lip movements.Lip error is the maximal L2 error of all lip vertices per frame, averaged over testing sequences.
- User studies: User studies favor FaceFormer over VOCA and MeshTalk for realistic facial animation and lip sync.The reported advantages are attributed to more expressive facial motions and fewer mouth-closure failures at /b/, /m/, and /p/ than VOCA.
- Ablation studies: Removing encoder self-attention, wav2vec initialization, the alignment bias, or the autoregressive decoder degrades synchronization, motion quality, expressiveness, or stability.The reported failures include incomplete mouth closure, out-of-sync or jittery lip motions, muted expressions, and less stable mouth motion.
- Ablation studies: The TB+PPE positional encoding strategy improves robustness on longer audio sequences, whereas Original PE can jitter and ALiBi can freeze facial expressions.The long-audio perceptual evaluation uses approximately 20-second TED clips, exceeding four times the average training-sequence length.
5. Discussion and Conclusion
FaceFormer is presented as an autoregressive transformer architecture for speech-driven 3D facial animation, using pre-trained speech representations, long-range audio dependencies, and tailored decoder attention. The authors also warn that such techniques could be misused and call for public awareness and responsible-use research.
- Contribution: FaceFormer uses an autoregressive transformer architecture for speech-driven 3D facial animation.Its encoder leverages self-supervised pre-trained speech representations, while inside self-attention captures long-range audio context dependencies.
- Contribution: Decoder attention modules with periodic position encoding are tailored for cross-modal alignment and generalization to longer sequences.
- Responsible Use: The authors warn that these techniques could potentially be misused to cause embarrassment.They hope to raise public awareness about potential misuse risks and encourage research on responsible technology use.
6. Implementation Details
FaceFormer uses a transformer-based encoder–decoder architecture with specified dimensions, attention heads, and periodic positional encodings. Training uses Adam, pretrained wav2vec 2.0 initialization, 100 epochs, and dataset-specific positional periods.
- Network architecture: The encoder applies TCN-based interpolation followed by 12 transformer encoder layers with model dimension 768 and 12 attention heads.The interpolation size is determined by the frequency of captured facial motion data.
- Network architecture: The decoder has one layer; its motion encoder and style embedding produce d outputs, while both biased attentions use 4 heads and model dimension d.Periodic positional encodings match the motion encoder dimension, and the feed-forward layer has dimension 2048.
- Training: Training uses Adam with learning rate 1e-4, initializes the encoder from pretrained wav2vec 2.0 weights, fixes only TCN parameters, and runs for 100 epochs.The periodic encoding period p is 25 for BIWI and 30 for VOCASET.
7. Baseline Methods
FaceFormer is compared with VOCA and MeshTalk on the BIWI and VOCASET datasets using dataset-specific training, testing, or provided-model protocols. MeshTalk’s decoder is modified to reduce overfitting caused by limited identity diversity.
- Comparison protocols: FaceFormer is compared with state-of-the-art VOCA and MeshTalk on both BIWI and VOCASET.VOCA uses its original implementation on BIWI and its provided trained model for VOCASET; MeshTalk is implemented and trained separately for each dataset.
- Decoder modification: MeshTalk’s UNet-style decoder is reduced from 7 to 3 fully-connected layers to avoid overfitting on BIWI and VOCASET.The modification addresses the limited number of identities in both datasets.
8. User Study
The user study used an AMT interface with video-pair comparisons and qualification checks designed to prevent random responses. Participants had to pass hidden tests and meet a prior-task completion requirement.
- Participant Qualification: One or two qualification videos were added to each HIT to discourage random option selection.Turkers who failed the hidden test could not submit their answers.
- Participant Recruitment: The recruitment requirement was that Turkers had finished over 5000 HI.
- Study Interface: Each AMT HIT contained four video pairs for user evaluation.The interface displayed one video pair at a time because of page limits.