Source-linked AI summary
CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior
Jinbo Xing, Menghan Xia, Yuechen Zhang, Xiaodong Cun, Jue Wang, Tien-Tsin Wong
TL;DR
Speech-driven 3D facial animation remains difficult because ambiguous speech-to-motion mappings and scarce paired audio-visual data can produce over-smoothed, less vivid motion. CodeTalker learns realistic discrete motion priors and uses a speech-conditioned temporal autoregressive code-query model; it reports superior qualitative and quantitative performance, including accurate lip synchronization and vivid expressions.
Problem
Speech-driven facial animation is ill-posed, and regression-based mappings can yield averaged, over-smoothed motions amid scarce paired audio-visual data.
Method
CodeTalker learns a discrete motion codebook by self-reconstructing real facial motions, then autoregressively predicts speech-conditioned tokens that query code sequences for motion reconstruction.
Results
CodeTalker outperforms existing state-of-the-art methods qualitatively and quantitatively, with accurate lip synchronization and vivid facial expressions.
Takeaways & Limitations
Discrete motion priors reduce cross-modal ambiguity while retaining realistic motion components for speech-driven 3D facial animation.
Takeaways & Limitations
The method assumes facial motions are independent of shapes, and its perceptual quality remains below ground truth because paired audio-visual data are scarce.
Abstract
from arXiv · showhide
Speech-driven 3D facial animation has been widely studied, yet there is still a gap to achieving realism and vividness due to the highly ill-posed nature and scarcity of audio-visual data. Existing works typically formulate the cross-modal mapping into a regression task, which suffers from the regression-to-mean problem leading to over-smoothed facial motions. In this paper, we propose to cast speech-driven facial animation as a code query task in a finite proxy space of the learned codebook, which effectively promotes the vividness of the generated motions by reducing the cross-modal mapping uncertainty. The codebook is learned by self-reconstruction over real facial motions and thus embedded with realistic facial motion priors. Over the discrete motion space, a temporal autoregressive model is employed to sequentially synthesize facial motions from the input speech signal, which guarantees lip-sync as well as plausible facial expressions. We demonstrate that our approach outperforms current state-of-the-art methods both qualitatively and quantitatively. Also, a user study further justifies our superiority in perceptual quality.
1. Introduction
Speech-driven 3D facial animation is ill-posed because one speech signal can correspond to multiple facial motions, making regression-based methods prone to over-smoothed expressions. CodeTalker instead queries a learned finite discrete motion space with a temporal autoregressive model to improve realism, lip synchronization, and expression vividness.
- Speech-driven facial animation is ill-posed, with multiple plausible facial outputs for each speech input.
- Regression-based speech-to-motion mapping encourages averaged motions, especially in upper-face regions weakly correlated with speech.
- CodeTalker represents facial motion using discrete primitives from a learned codebook, reducing uncertainty in cross-modal mapping.The codebook is learned through self-reconstruction of real facial motions and stores realistic motion priors.
- The proposed model predicts motion feature tokens autoregressively, queries discrete code sequences, and reconstructs facial motion from speech.Its temporal recursion models history motions while cross-modal alignment supports synthesis.
- CodeTalker outperforms existing state-of-the-art methods qualitatively and quantitatively, with reported advantages in lip accuracy and facial-expression vividness.
2. Related Works
Related work includes linguistics-based systems that map phonemes to visual counterparts and learning-based systems that learn speech-to-face mappings from data. CodeTalker extends discrete-prior learning to speaker-independent 3D facial animation by using context-rich discrete motion primitives.
- Linguistics-based methods: Linguistics-based methods construct phoneme-to-viseme rules for synchronized mouth movements but involve complex procedures and do not principledly animate the entire face.
- Learning-based methods: Learning-based methods use data-driven models for speech-driven facial animation, including convolutional networks, sliding-window models, and emotion databases.
- Related 3D facial animation: MeshTalk disentangles audio-correlated and uncorrelated facial information with a categorical latent space, but its limited expressiveness makes animation quality unstable in scarce-data settings.
- Discrete prior learning: Discrete prior representations preserve detail in image restoration and have also been explored for face motion generation, speech gesture synthesis, and speech enhancement.
- Discrete prior learning: CodeTalker learns context-rich discrete motion primitives for speaker-independent 3D facial animation, differing from prior face-motion work in its motion representation.
3. Method
CodeTalker represents facial motion with a learned discrete codebook prior, then predicts motion codes autoregressively from speech and style to reconstruct facial motions. This design reduces mapping ambiguity while retaining context-rich motion detail and supports generic, lip-synchronized animation.
- 3.2. Speech-Driven Motion Synthesis: The formulation synthesizes sequential facial motions that can animate an arbitrary neutral face mesh as a lip-synchronized talking face.Each motion frame is added to the neutral-face template to produce the animated mesh sequence.
- 3.1. Discrete Facial Motion Space: The method first learns a context-rich codebook and decoder through VQ-VAE self-reconstruction of realistic facial motions.Encoded motions are quantized to nearby codebook entries and decoded back into reconstructed motions.
- 3.1. Discrete Facial Motion Space: Discrete codebook items act as motion primitives representing facial-component movements across spatial and temporal dimensions.The discrete space reduces mapping ambiguity while preserving expressiveness through context-rich latent representations.
- 3.1. Discrete Facial Motion Space: The VQ-VAE objective combines motion reconstruction with two code-level losses that reduce distances between embedded features and codebook items.The first term reconstructs facial motion; the latter two update the codebook and encoder-related representations.
- 3.1. Discrete Facial Motion Space: Unlike Learn2Listen’s speaker-specific expression-sequence codes, CodeTalker uses a generic vertex-based codebook with motion primitives for arbitrary facial motions.The generic codebook absorbs cross-character motions and aims to preserve vivid facial details beyond 3DMM coefficient representations.
- 3.2. Speech-Driven Motion Synthesis: A temporal autoregressive model conditions on speech, style, and past motions to predict motion tokens recursively, quantize them, and decode facial motions.Causal self-attention models motion history, while cross-modal attention aligns audio and motion modalities; the pretrained codebook and decoder remain frozen during this training.
4. Experiments
Experiments evaluate CodeTalker on BIWI and VOCASET using lip synchronization, upper-face dynamics, visual comparisons, style interpolation, and user preference. Across these evaluations, the method produces accurate lip motion, stronger and richer facial dynamics, interpolated speaking styles, and favorable perceptual quality, while ablations examine representation and codebook design.
- Datasets and Implementations: Experiments use BIWI and VOCASET, comparing CodeTalker with VOCA, MeshTalk, and FaceFormer under established evaluation settings.BIWI-Test-A supports quantitative and qualitative evaluation, while BIWI-Test-B is primarily suitable for qualitative evaluation; VOCA-Test follows established splits for fair comparison.
- Quantitative Evaluation: Lip vertex error measures synchronization, while upper-face dynamics deviation measures differences between predicted and ground-truth upper-face motion variation.Lip vertex error averages the maximal L2 error over lip vertices and frames; FDD uses temporal variation statistics over upper-face vertices.
- Quantitative Evaluation: CodeTalker achieves lower lip vertex error and the best upper-face dynamics deviation on BIWI-Test-A among the compared methods.The reported results indicate more accurate lip-synchronized movements and higher consistency between predicted and ground-truth upper-face dynamics.
- Qualitative Evaluation: Visual comparisons show more accurate lip articulation, stronger facial movements, and broader temporal dynamics than competing methods.Examples include proper mouth closure for /b/ and accurate pout shapes for “waterproof” and “shaving,” while temporal statistics show stronger movements and richer variation.
- Qualitative Evaluation: Interpolating two style vectors produces smooth transitions in mouth amplitudes and can synthesize new speaking styles for matching unseen performances.The interpolation combines style vectors associated with large and slight lip articulations.
- User Study: User studies on BIWI-Test-B and VOCA-Test favor CodeTalker over competitors, associating preference with expressive motions, accurate lip shapes, and synchronized mouth movements.The study reports over 40% preference even against recorded ground truth on VOCA-Test, despite a gap between predictions and recorded performance.
- Ablation Studies: Ablations show that motion-based codebooks outperform shape-entangled representations, while codebook temporal length and component count affect reconstruction and lip vertex error.Longer temporal units degrade reconstruction, whereas increasing the number of face components helps below H = 8 but performs worse at H ≥ 8.
5. Discussion and Conclusion
CodeTalker improves lip synchronization and facial-expression vividness by querying a discrete motion space, while remaining limited by data scarcity, shape-independence assumptions, and training-distribution coverage.
- The discrete code-query formulation promotes motion synthesis quality against ambiguity and achieves accurate lip sync with vivid facial expressions.
- The method assumes facial motions are independent of shapes, whose rationality requires further study.
- Scarce paired audio-visual data leaves overall perceptual quality below ground truth.
- The generic motion prior follows the training-set motion distribution, which may differ from real-world facial motions.
A.1. Hyper-parameters of Codebook
The supplemental discussion identifies the parameter setting selected for high-quality BIWI results using ground-truth evaluation on BIWI-Test-A.
- For BIWI, ground-truth evaluation on BIWI-Test-A selects P = 1 and H = 8 for high-quality results.
A.2. Network Architecture
The supplement documents CodeTalker’s network architectures to improve reproducibility.
- Detailed architectures are provided for facial motion-space learning and speech-driven motion synthesis.
B.1. Instance Normalization in Self-reconstruction Learning
Instance Normalization improves CodeTalker’s self-reconstruction and helps preserve accurate lip-motion amplitudes in reconstruction and speech-driven synthesis.
- Instance Normalization computes mean and variance across temporal dimensions independently for each channel within each sample.
- Instance Normalization improves self-reconstruction performance and training stability, whereas Batch Normalization is avoided because of small mini-batches.
- With Instance Normalization, reconstructed lip amplitudes more closely match ground truth than without normalization.
- The Instance-Normalization codebook also benefits speech-driven synthesis conditioned on a randomly sampled speaking style.
- The supplement describes the network architecture and reports that Table 5 measures reconstruction error on VOCA-Test and BIWI-Test-A.
- The authors identify the rationality and effect of Instance Normalization as requiring further study.
B.2. Alternative Data Flow and Supervision
The alternative decoder predicts discrete facial-motion codes autoregressively, retrieves codebook entries, and reconstructs motions with a fixed decoder. Its token-prediction framework performs worse than CodeTalker for lip-motion accuracy, even with additional constraints.
- Alternative data flow: The alternative cross-modal decoder uses past motion codes to autoregressively predict discrete code sequences.The framework omits style and audio inputs in the illustrative data flow.
- Alternative data flow: Predicted codes retrieve entries from the learned codebook Z and are decoded into facial-motion sequences through the fixed decoder D.Ground-truth codes are generated by the pretrained encoder E.
- Supervision: Categorical cross-entropy supervises the predicted code sequence against the encoder-generated ground-truth code sequence.The loss is applied to token predictions in the discrete motion space.
- Results: CodeTalker achieves lower average lip error than the alternatives, indicating greater lip-movement accuracy.The alternative trained with Lce alone cannot converge well; adding regression and motion constraints improves learning but remains limited.
C. User Study
The user study presents participants with paired facial-animation videos and asks them to compare lip movements and facial expressions. Each participant evaluates 24 video pairs under a controlled viewing procedure.
- Study design: Each participant evaluates 24 video pairs in the designed user-study interface.The study is designed to take approximately 5–10 minutes, based on repeated five-second video viewing.
- Study design: Participants judge the paired videos twice using separate questions about lip movements and facial expressions.The passage begins the first question as a direct comparison of the two faces’ lips.
- Study design: Results completed in less than two minutes are filtered out to reduce the impact of random selection.The filtering is applied per participant before analyzing comparison responses.
D. Video Comparison
The video comparison evaluates CodeTalker on varied speech clips and reports natural, plausible animations with synchronized lips and more vivid motion than competing methods.
- Video comparison: The comparison uses audio from TED and TEDx videos, VOCASET and BIWI datasets, and previous methods’ supplementary videos.The evaluation covers multiple sources of speech audio.
- Video comparison: CodeTalker synthesizes natural and plausible facial animations with well-synchronized lip movements across the demonstrated audio clips.The supplemental video is provided for qualitative demonstration and comparison with competitors.
- Video comparison: Compared with VOCA, MeshTalk, and FaceFormer, CodeTalker produces more vivid and realistic facial motion rather than over-smoothed motion.The comparison is presented as a qualitative video result.