Source-linked AI summary
VisemeNet: Audio-Driven Animator-Centric Speech Animation
Yang Zhou, Zhan Xu, Chris Landreth, Evangelos Kalogerakis, Subhransu Maji, Karan Singh
TL;DR
The paper addresses the challenge of producing editable, animator-centric speech animation directly from audio without language-specific transcripts. It proposes a three-stage LSTM pipeline that predicts phoneme groups, landmarks, and JALI motion curves, achieving near-real-time audio-driven animation and reported robustness across speakers and languages. The method is limited to controlling the lower face in its current implementation.
Problem
Existing speech-animation workflows and transcript-based systems limit editability, language independence, or animator-centric control.
Method
A three-stage LSTM network predicts phoneme groups and facial landmarks from audio, then combines them with audio features to produce JALI viseme motion curves.
Results
The architecture produces animator-centric speech animation directly from audio with near-real-time performance at 120ms lag and supports diverse speakers and languages.
Takeaways & Limitations
Jointly optimized intermediate phoneme-group and landmark stages support a compact, animator-editable audio-to-animation pipeline.
Takeaways & Limitations
The current implementation controls only the lower part of the face; extending control to the upper face is future work.
Abstract
from arXiv · showhide
We present a novel deep-learning based approach to producing animator-centric speech motion curves that drive a JALI or standard FACS-based production face-rig, directly from input audio. Our three-stage Long Short-Term Memory (LSTM) network architecture is motivated by psycho-linguistic insights: segmenting speech audio into a stream of phonetic-groups is sufficient for viseme construction; speech styles like mumbling or shouting are strongly co-related to the motion of facial landmarks; and animator style is encoded in viseme motion curve profiles. Our contribution is an automatic real-time lip-synchronization from audio solution that integrates seamlessly into existing animation pipelines. We evaluate our results by: cross-validation to ground-truth data; animator critique and edits; visual comparison to recent deep-learning lip-synchronization solutions; and showing our approach to be resilient to diversity in speaker and language.
1 INTRODUCTION
Realistic speech animation is important but difficult to produce with existing workflows: keyframing is expressive yet laborious, while performance capture is difficult to edit. VisemeNet addresses this gap with a three-stage audio-to-animation network designed for animator-centric, near-real-time and language-agnostic speech animation.
- Realistic facial animation matters across entertainment, medicine, education, and assistants because subtle flaws can undermine audience trust and empathy.
- Keyframing is expressive and editable but laborious, whereas performance capture is easier to record yet difficult to edit or refine.
- Transcript-based pipelines add complexity and language dependence, while audio-only approaches can avoid phonetic transcripts.
- The proposed network predicts phoneme groups, facial landmarks, then JALI parameters and sparse speech motion curves from audio.
- The system targets animator-centric speech animation directly from audio with near-real-time performance at 120ms lag.
2 RELATED WORK
Prior speech-animation methods trade off editability, realism, language dependence, or compatibility with animator workflows. VisemeNet instead maps audio through intermediate phoneme-group and landmark representations to sparse, compact animator-centric motion curves in a jointly optimized pipeline.
- Procedural: Procedural animation maps phonemes to visemes but must handle contextual mappings, co-articulation, and viseme motion profiles.
- Performance-capture: Performance-capture transfers recorded facial motion effectively, but the resulting animation is not inherently animator-centric or easily editable.
- Data-driven: Data-driven methods stitch facial-animation examples from corpora, limiting scope to available data and producing non-animator-centric output.
- Deep learning-based speech animation: Transcript-based deep-learning methods introduce language dependence and output face-space representations rather than compact animator-centric viseme curves.
- Deep learning-based speech animation: Audio-to-mesh methods avoid transcripts but produce outputs poorly suited to current animation practice and animator-centric editing.
- VisemeNet: The proposed pipeline jointly optimizes all stages, avoiding independently fitted rig predictions that can break perceptual constraints.
3 ALGORITHM DESIGN
VisemeNet combines psycho-linguistic structure with LSTM-based prediction to convert audio into compact JALI speech controls. It uses phoneme groups, facial landmarks, audiovisual transfer learning, and multi-task training to support style-aware animator-editable animation.
- The system targets high-quality, animator-editable, style-aware, language-agnostic, real-time speech animation from audio.
- Its architecture predicts sparse viseme values together with jaw and lip parameters over time for FACS-like face rigs.
- Phoneme group prediction: The phoneme-group stage recognizes 20 visually relevant IPA groups from audio without text or a phonetic transcript.
- Speech style prediction: The landmark stage captures speech-style cues because jaw and lip parameters correlate strongly with jaw and lip landmark motion.
- Viseme prediction: The viseme stage combines phoneme groups, landmarks, and audio to predict JALI-based viseme representations using multimodal speech data.
- Transfer learning from audiovisual datasets: Audiovisual corpora provide landmarks and aligned transcripts that support transfer learning despite lacking facial rig parameters.
- Multi-task learning jointly trains phoneme-group, landmark, viseme, and rig-parameter objectives, which the authors found important for high performance.
- LSTMs provide memory for representing speech context needed to capture co-articulation from audio.
4 NETWORK ARCHITECTURE
The network converts audio into animator-centric JALI rig outputs through staged phoneme-group, landmark, and viseme processing. LSTM-based stages use temporal audio context and intermediate representations to predict sparse, style-sensitive facial controls.
- 4 NETWORK ARCHITECTURE: The architecture first processes audio through phoneme-group and landmark stages, then combines their predictions with audio features for viseme prediction.The final stage outputs JALI-based rig parameters and controls that animate the face.
- Phoneme group stage: The phoneme-group stage uses a temporal audio window to predict phoneme-group probabilities, introducing a 120 ms real-time lag.The window includes past and future frames, while feature extraction and network inference take about 1 ms per frame.
- 4 NETWORK ARCHITECTURE: Three layers of unidirectional LSTM units hierarchically update 256-dimensional memory states from frame-level audio features.The architecture uses recurrent memory to represent temporal audio information.
- Landmark stage: The landmark stage predicts 38 sparse 2D jaw, lip, and nose landmarks from the same 1560-dimensional frame feature vector.The landmarks approximate an average face rather than the morphology of a particular face.
6 EVALUATION
The evaluation uses BIWI to compare the proposed method and alternatives under neutral and expressive speech conditions. Artist-provided JALI motion curves provide ground truth, while leave-one-speaker-out evaluation addresses the dataset’s limited speaker count.
- Evaluation: The study evaluates the proposed method and alternatives quantitatively and qualitatively, focusing the section primarily on quantitative results.Qualitative results and comparisons are referred to in the accompanying video.
- Methodology: BIWI provides artist-generated JALI motion curves for comparing predicted rig parameters with ground truth.The dataset includes neutral and expressive renditions, including anger, sadness, fear, nervousness, and excitement.
- Methodology: Leave-one-out evaluation holds out each of BIWI’s 14 speakers, pre-trains on the remaining audiovisual data, and jointly trains on the other 13 annotated speakers.This protocol is used because the JALI-annotated dataset contains only 14 speakers.
14 training
The evaluation compares the full architecture with degraded alternatives using activation and motion-curve measures across neutral and expressive speech. Tests use speaker-independent splits and assess whether intermediate stages and recurrent modeling improve performance.
- Evaluation setup: Test splits always use speakers not observed during training to assess speaker-independent generalization.The authors consider this more practically useful than training and testing on the same speaker.
- Evaluation measures: Precision measures correct predicted rig activations among predicted activations, while recall measures correct activations among ground-truth activations.Both measures are ideal at 100%.
- Evaluation measures: Motion-curve differences measure absolute discrepancies in active viseme, co-articulation, and JALI parameters, with inactive parameters excluded because the curves are sparse.Normalized parameter differences can be interpreted as percentages.
- Results: The full method achieves the best performance across all evaluation measures and both neutral and expressive speech styles.The comparison reports precision, recall, and motion-curve differences averaged over all test splits.
- Ablations: Removing the phoneme-group or landmark stages causes large or noticeable performance drops, showing that both intermediate representations contribute to viseme prediction.The phoneme-group stage is described as crucial for predicting correct visemes.
- Ablations: Replacing LSTMs with fully connected sliding-window modules causes a large performance drop.The alternative architectures were trained with corresponding losses, matched training sets, hyper-parameter tuning, and test splits.
- Comparison limits: Quantitative comparison with prior methods is unavailable because their test rigs are not FACS-enabled and their implementations are not public.The paper instead provides qualitative comparisons with previous works.
7 CONCLUSION
The paper presents an animator-centric deep-learning system that maps audio to speech motion curves. It identifies broader extensions, including learned audio features, a discriminator, and control of the upper face.
- Conclusion: The approach maps audio to speech motion curves for animator-centric facial animation.The stated future directions build on this lower-face animation system.
- Future work: The current implementation uses hand-engineered audio features, which the authors propose replacing with learned features to improve performance.The proposed change is presented as future work rather than a demonstrated result.
- Limitations: The method currently controls only the lower face; learning upper-face control, such as eye movement, without explicit supervision is proposed as future work.This is the paper's clearest scope boundary.
- Future work: The authors suggest discriminator-based quality assessment as another way to boost the viseme generator's performance.They connect this proposal to cGAN-based approaches for image and shape synthesis.
- Implications: The authors argue that combining animator-centric techniques with deep learning could leave animators more time for creative and nuanced character expression.They frame this as potential impact on facial animation practice in film and game studios.