VisemeNet: Audio-Driven Animator-Centric Speech Animation

• Realistic facial animation serves entertainment, medicine, and education, but subtle flaws can undermine audience trust and empathy, especially as rendered realism increases.Source: §1 · p. 1
• Keyframing is expressive and editable but laborious, while performance capture is easy to record but difficult to refine and may involve visually inexpressive actors.Source: §1 · p. 1
• JALI offers animator-centric procedural lip-sync with audio and transcript, while deep audio-to-face approaches poorly fit animator workflows.Source: §1 · p. 1
• Transcript-and-alignment pipelines using and are error-prone and language/lexicon-specific, whereas Karras et al. show audio can predict faces agnostic of language.Source: §1 · p. 1
• The network predicts phoneme-groups, uses facial-landmark locations as cues to speech style, then maps both to JALI parameters and sparse motion curves encoding animator or speaker style.Source: §1 · p. 2
• VisemeNet produces animator-centric speech animation directly from audio in near real time, with 120 ms lag.Evaluation includes ground-truth cross-validation, professional-animator critique and editing, comparison with non-animator-centric systems, and testing language agnosticism with diverse speech data.Source: §1 · p. 2

Procedural

• Procedural systems map phonemes to visemes using context-sensitive mappings, co-articulation models, and viseme profiles; JALI produces compact animator-friendly curves from phonetic input.Viseme profiles describe attack, apex, sustain, and decay over time.Source: §2 · p. 2

Performance-capture

• Performance capture transfers a performer’s motion to a digital face and can produce high-quality general facial animation, but is limited by the actor’s performance and is difficult to edit or refine.Capture has become more powerful and mainstream with cameras, depth sensors, and reconstruction techniques, including 3D face modeling from a single image.Source: §2 · p. 2

Data-driven

• Data-driven methods stitch facial-animation segments from a corpus to match speech, using morphable, hidden Markov, and active appearance models, but are limited by available data and are not animator-centric.Examples include,,, and.Source: §2 · p. 2

Deep learning-based speech animation

• Unlike Taylor et al. [2017], which requires a transcript and predicts an 8-dimensional face-space, this method maps audio directly to sparse, low-dimensional viseme curves tied to speech.Taylor et al.’s output can be retargeted and post-edited, but lacks a compact mapping to animator-centric viseme curves or facial action units, making spoken content or style harder to refine.Source: §2 · p. 2
• Karras et al. [2017] predicts 3D mesh vertices rather than animator-centric controls; this method avoids separate rig-fitting steps and trains its full pipeline to minimize animation error.Separate fitting can create temporal discontinuities from rig redundancy, while independently editing correlated curves can violate perceptual constraints and later fitting can exacerbate accumulated vertex-position errors.Source: §2 · pp. 2–3
• Suwajanakorn et al. [2017] synthesize Barack Obama’s speaking video from audio and target video; their successful use of LSTMs encouraged this method’s choice of LSTMs for its network stages.Their task differs from this paper’s speech-animation problem.Source: §2 · p. 3
• The proposed three-stage network segments audio into phonetic groups, predicts facial landmarks for speech style, then combines these with audio features to produce viseme curves, training all stages jointly.It uses multi-task learning to minimize errors in phonetic groups, landmarks, visemes, co-articulation, and speech-style parameters; training combines clips with ground-truth phonemes, tracked landmarks, and 3D animations with ground-truth visemes.Source: §2 · p. 3

Phoneme group prediction

• The phoneme-group stage recognizes 20 animation-relevant IPA groups directly from audio without transcripts, simplifying prediction and reducing linguistic-context sensitivity.The authors report lower performance when off-the-shelf audio-to-text methods extract individual phonemes that are then used to predict visemes.Source: §3 · p. 3

Speech style prediction

• Because phoneme groups omit vocal delivery, the landmark stage predicts jaw and lip positions from audio to capture style information associated with facial landmarks.Jaw and lip parameters capture differences in delivery, such as conversational versus screamed speech.Source: §3 · p. 3

Viseme prediction

• The viseme stage combines phoneme-group, jaw, and lip predictions with audio to produce JALI-model visemes; viseme and co-articulation values can animate standard FACS-based rigs and support editing.The representation uses intensity values for 20 visemes and 9 co-articulation rules, while JAW and LIP parameters capture speaking style and provide rig control over speech style.Source: §3 · p. 3

Transfer learning from audiovisual datasets

• Transfer learning pretrains phoneme-group and landmark stages on audiovisual corpora, then jointly trains the network with a smaller rig-parameter animation dataset, improving generalization over training only on that small dataset.Video-derived facial landmarks train the landmark stage, and transcript-aligned phonemes train the phoneme-group stage; the authors also report that multi-task learning across prediction objectives is important for high performance.Source: §3 · pp. 3–4

Memory-enabled networks

• LSTM units model speech context and co-articulation, enabling near-real-time viseme motion-curve prediction with a 120 ms or 12-frame lag on modern GPUs.The authors note that speech context can be difficult to infer from isolated audio fragments, even for human listeners.Source: §3 · p. 4

NETWORK ARCHITECTURE

• Three LSTM-based stages predict phoneme groups and facial landmarks from audio, then combine those representations with audio features to produce viseme controls.Each stage uses recurrent units to encode context and fully connected layers to decode predictions.Source: §4 · p. 4

Input audio representation

• The input is a 65-dimensional audio feature vector—13 MFCCs, 26 Mel Filter Bank features, and 26 Spectral Subband Centroid features—extracted at 100 FPS using 25 ms windows.The same features feed all three network stages.Source: §4 · p. 5

Phoneme group stage

• The phoneme-group stage processes a 24-frame, 240 ms audio window with three unidirectional LSTM layers, yielding real-time inference with a 120 ms lag.The 240 ms window approximately matches the average phoneme duration in normal speech; a bidirectional LSTM offered no noticeable difference.Source: §4 · p. 5

Landmark stage

• The landmark stage predicts 38 2D jaw, lip, and nose landmarks for an average face; it shares LSTM layers with phoneme prediction and helps capture speech style.The stages share layers because phonetic groups and lower-face shape are correlated.Source: §4 · p. 5

Viseme stage

• The viseme stage uses phoneme activations, landmark displacements, and audio features to predict 29 continuous rig controls, 29 binary activation indicators, and a 2D JALI viseme field.Separate LSTM layers predict each output type; binary indicators address the sparsity of rig-control activations.Source: §4 · p. 6

Training procedure

• The network is pre-trained on 15 hours of audiovisual data, then jointly fine-tuned on one hour of audio with animator-created JALI rig curves.The JALI-annotated dataset covers the 14 BIWI speakers.Source: §5 · pp. 6–7

Datasets and preprocessing

• GRID, SAVEE, and BIWI recordings provide transcripts, audio, and video; preprocessing extracts normalized facial landmarks and aligns phonemes to audio using transcripts.The JALI rig-curve dataset was created for BIWI.Source: §5 · pp. 6–7

Pre-training

• Pre-training combines phoneme-group classification, landmark-coordinate L1 regression, and a smoothness loss that penalizes abrupt landmark motion.The loss weights are 0.75 for classification, 0.25 for landmark regression, and 0.1 for smoothness.Source: §5 · p. 7

Joint training

• Joint training adds rig-parameter activation classification, active-parameter regression, JALI-field regression, and smoothness losses while retaining phoneme and landmark objectives.The losses are coupled because viseme-branch predictions depend on predicted phonemes and landmarks.Source: §5 · p. 7

Thresholding activation probabilities and hold-out validation

• Rig-parameter activation thresholds are tuned per parameter on a 10% hold-out set to optimize precision and recall rather than fixed at 0.5.The same hold-out procedure selects loss weights; evaluating parameter values during inactive periods degraded prediction quality.Source: §5 · p. 8

Implementation and running times

• Pre-training takes 30 hours and joint training 15 hours on a TitanX GPU, while test-time audio processing and inference run at 100 FPS with 120 ms lag.The network is implemented in TensorFlow.Source: §5 · p. 8

Methodology

• Evaluation uses BIWI’s 14 speakers, each speaking the same sentence in neutral and expressive styles that include several emotions.Fourteen leave-one-speaker-out splits test speakers excluded from training; training uses the other 13 speakers, with pre-training on GRID, SAVEE, and BIWI data excluding the held-out speaker.Source: §6 · p. 8

Quantitative evaluation measures

• The measures assess rig-parameter activation precision and recall, plus average absolute motion-curve differences for active ground-truth or predicted parameters.Motion-curve differences exclude inactive parameters because the curves are sparse; normalized rig values make the differences interpretable as percentages.Source: §6 · pp. 8–9

Quantitative Comparisons

• The full method performs best across all reported evaluation measures for both neutral and expressive speech styles.The comparisons cover rig-control activation precision and recall and motion-curve differences; the supplied passages do not provide table-cell values.Source: Table 1 · p. 9, Table 2 · p. 9, §6 · p. 9
• Removing the phoneme-group stage causes a large performance drop, while removing landmarks causes a noticeable drop; GRUs and sliding-window networks also underperform LSTMs.Transfer learning and weight sharing between the landmark and phoneme stages improve performance; the ASR-based alternative also performs worse and is language-dependent, unlike the IPA-based approach.Source: §6 · pp. 9–10

Qualitative comparisons

• The video compares the method with prior speech-animation systems and demonstrates animation for different languages.Quantitative comparison with prior systems is not possible because their rigs are not FACS-enabled and their implementations are unavailable; unlike this method, they do not produce editable animator-centric viseme curves or facial action units.Source: §6 · p. 10

• The approach maps audio to speech motion curves using an animator-centric deep-learning method.Source: §7 · p. 10
• Future work could replace hand-engineered audio features with learned features to improve performance.Source: §7 · p. 10
• A discriminator network could assess generated-animation quality and use its predictions to improve the viseme generator, as in cGAN-based image and shape synthesis [Isola et al. 2016].Source: §7 · p. 10
• The method currently drives only the lower face; learning upper-face control, such as eye motion, without explicit supervisory signals is a future direction.Source: §7 · p. 10
• The authors argue that combining animator-centric techniques with deep learning could reshape facial animation in film and game studios, enabling animators to focus on creative, nuanced expression.Source: §7 · p. 10