Source-linked AI summary
Realistic Speech-Driven Facial Animation with GANs
Konstantinos Vougioukas, Stavros Petridis, Maja Pantic
TL;DR
The paper addresses realistic talking-head synthesis requiring detailed faces, synchronized lip movements, and plausible expressions from speech. It proposes an end-to-end temporal GAN using a still image and audio, producing natural-looking videos with synchronized speech and spontaneous facial gestures.
Problem
Realistic talking-head generation must simultaneously produce high-quality faces, audio-synchronized lip movements, and plausible expressions, while existing CGI approaches require substantial equipment and labor.
Method
An end-to-end temporal GAN generates talking-head videos from a single still image and audio, using separate discriminators for frame detail, synchronization, and facial movements, plus temporally coherent noise for spontaneous expressions.
Results
The generated videos produce detailed frames, improve word error rate over static and Speech2Vid baselines, support spontaneous blinks and head and brow movements, and appear natural in a Turing test.
Takeaways & Limitations
Separating synchronization from expression modeling and using sequence-level adversarial training supports speech-aligned facial motion alongside spontaneous gestures and reflected speaker emotion.
Takeaways & Limitations
The method is limited to well-aligned frontal faces, motivating future work on in-the-wild conditions.
Abstract
from arXiv · showhide
Speech-driven facial animation is the process that automatically synthesizes talking characters based on speech signals. The majority of work in this domain creates a mapping from audio features to visual features. This approach often requires post-processing using computer graphics techniques to produce realistic albeit subject dependent results. We present an end-to-end system that generates videos of a talking head, using only a still image of a person and an audio clip containing speech, without relying on handcrafted intermediate features. Our method generates videos which have (a) lip movements that are in sync with the audio and (b) natural facial expressions such as blinks and eyebrow movements. Our temporal GAN uses 3 discriminators focused on achieving detailed frames, audio-visual synchronization, and realistic expressions. We quantify the contribution of each component in our model using an ablation study and we provide insights into the latent representation of the model. The generated videos are evaluated based on sharpness, reconstruction quality, lip-reading accuracy, synchronization as well as their ability to generate natural blinks.
1 Introduction
Realistic talking-head synthesis must jointly address detailed faces, audio-synchronized lip motion, and plausible expressions, but prior approaches are often subject dependent or expression-poor. The paper proposes a temporal GAN that generates videos from one still image and speech while evaluating visual quality, synchronization, identity, expressions, and component contributions.
- Realistic talking-head generation requires high-quality faces, synchronized lip movements, and plausible expressions because viewers detect subtle motion and audio-visual abnormalities.
- Most prior systems map audio features to visual features and use computer graphics or person-specific databases, creating subject dependence and transfer overhead.
- Many subject-independent methods focus on mouth motion while neglecting facial expressions, whose absence can make generated videos visibly synthetic.
- Static frame-generation methods ignore facial-motion dynamics, while whole-sequence or batch generation can introduce lag and require fixed-length training sequences.
- The proposed temporal GAN generates talking-head videos from speech and one still image, using separate temporal discriminators for synchronization and facial movements.
- The study evaluates reconstruction, sharpness, lip-reading, identity preservation, synchronization, blinks, and discriminator contributions across GRID, TCD TIMIT, CREMA-D, and LRW.
2 Related Work
Related work progresses from hand-designed or person-specific visual representations toward neural and GAN-based models for speech-driven facial synthesis. Persistent limitations include speaker dependence, blurry or static outputs, simplified facial dynamics, and fixed-length video constraints.
- Early approaches used acoustic-visual correlations, graph searches, interpolation, and HMMs to model speech-linked facial or mouth-shape sequences.
- Feature-based systems transform phonemes, MFCCs, or audio into facial descriptors, meshes, mouth shapes, or retrieved frames using DNNs, CNNs, and RNNs.
- Speaker-specific mouth shapes and 3D meshes require retraining or re-targeting when adapting to new faces, motivating speaker-independent representations.
- A subject-independent CNN system produces blurry frames under pixel-level L1 loss and mostly static faces because the objective does not encourage spontaneous expressions.
- Video GANs use temporal convolutions, recurrent generators, motion-content latent spaces, or sliding windows, but fixed-length handling and latent consistency remain design concerns.
- GAN-based cross-modal systems extend generative modeling to text-to-video and audio-to-video conversion using VAEs, 3D CNNs, encoder-decoders, and attention.
3 Speech-Driven Facial Synthesis
The proposed speech-driven facial synthesis architecture combines a temporal generator with multiple discriminators. Each discriminator evaluates a different aspect of the generated sequence, supporting more comprehensive realism assessment.
- The architecture consists of a temporal generator and multiple discriminators that evaluate generated sequences from different perspectives.
3.1 Generator
The generator combines a still image, centered overlapping audio frames, and temporally dependent noise to produce identity-preserving talking-head frames. Its encoder-decoder design separates identity, speech content, and spontaneous-expression representations before decoding each frame.
- The generator accepts one still image and overlapping 0.2-second audio frames centered on corresponding video frames.
- The generator uses a Content Encoder, Identity Encoder, Frame Decoder, and Noise Generator.
- Its latent representation separates speaker identity, audio content, and spontaneous facial expressions into components produced by different modules.
- The audio encoder and one-layer GRU produce a 256-element content encoding from encoded audio frames.
- Temporally dependent noise models spontaneous expressions, while skip connections between the Identity Encoder and Frame Decoder preserve subject identity.
3.2 Discriminators
Three discriminators evaluate generated videos from complementary perspectives: frame realism and identity, sequence-level motion, and audio-visual synchronization. The synchronization discriminator is trained with aligned, shifted, and generated audio-video pairs.
- The Frame, Sequence, and Synchronization Discriminators respectively assess face reconstruction, cohesive natural movements, and audio-visual synchronization.
- The Frame Discriminator judges individual frames as real or synthetic and conditions on the still image to enforce speaker identity.
- The Sequence Discriminator uses spatio-temporal CNN features followed by a GRU to classify entire videos as real or synthetic.
- The Synchronization Discriminator compares audio and video embeddings from 0.2-second snippets using their Euclidean distance.
- Training synchronization discrimination includes real synchronized pairs, real pairs with shifted audio, and fake videos with matching audio.
3.3 Training
Training combines adversarial objectives for frame, synchronization, and sequence discriminators with a lower-half L1 reconstruction loss. The reconstruction term supports mouth accuracy while its spatial restriction avoids discouraging facial expressions.
- The total adversarial loss aggregates the Frame, Synchronization, and Sequence discriminator losses with separate weights.
- The L1 reconstruction loss is applied only to the lower half of each image because applying it more broadly discourages facial expressions.
- The model loss combines adversarial and reconstruction terms, using λrec = 600, λimg = 1, λsync = 0.8, and λseq = 0.2.
- Training stops after five validation epochs without improvement in audio-visual synchronization.
- The generator and Frame Discriminator use learning rate 0.0001, while the Sequence and Synchronization Discriminators use 10^-5 to avoid discriminator overtraining and instability.
4 Datasets
The experiments use GRID, TCD TIMIT, CREMA-D, and LRW, with dataset-specific speakers, utterances, emotions, splits, and preprocessing choices.
- Datasets: Experiments use GRID, TCD TIMIT, CREMA-D, and LRW datasets.GRID contains 33 speakers; TCD TIMIT 59; CREMA-D 91 actors with varied ages, races, emotions, and intensities.
- Dataset organization: Table 1 lists the subject IDs tested for each dataset, while Table 2 reports samples and video hours across the splits.
- Splits: GRID speakers are divided into training, validation, and test sets using a 50%−20%−30% split.
- Splits: CREMA-D uses 70%−15%−15% training, validation, and test splits.
- Preprocessing: LRW uses its recommended splits but limits training faces to nearly frontal poses selected with pose estimation.
5 Metrics
The evaluation combines frame-quality, content, synchronization, and expression metrics, including a blink detector validated on manually annotated videos.
- Frame quality: Frame quality is assessed with PSNR, SSIM, and CPBD, where larger values indicate better quality for these measures.PSNR and SSIM measure reconstruction, while CPBD evaluates blur through image edges.
- Content: Content evaluation measures identity preservation with ACD and spoken-word accuracy with word error rate.
- Audio-visual synchrony: SyncNet estimates audio-visual offset by minimizing audio-video encoding distance over sliding 0.2-second windows.Offset is measured in frames, with positive values indicating that audio leads video.
- Expressions: Blink analysis uses eye aspect ratio, detecting a blink at a sharp EAR drop and estimating its boundaries from neighboring peaks.The landmarks and detection procedure are illustrated in Figures 7 and 8.
- Expression statistics: Real-video blink distributions report blink count and median duration, providing comparison targets for generated videos.
- Detector validation: The blink detector achieves 80% accuracy and detects blink start and end points with an average error of 1.75 frames.It uses a strict EAR-drop threshold to maintain very high precision; evaluation used 50 randomly selected GRID validation videos and manual annotations.
6 Experiments
The temporal GAN generates sharp, synchronized talking-head videos with identity preservation and spontaneous expressions, while temporal modeling improves coherence and blink realism.
- Efficiency: Inference averages 7ms per frame on a GPU, allowing real-time applications; 75 frames can be synthesized in 0.5s.CPU processing increases frame and sequence generation times to 1s and 15s, respectively.
- Identity: The identity distance remains close to real-video values, indicating that the model preserves speaker identity throughout generated sequences.The identity encoding separates test-set frames by gender in the t-SNE visualization.
- Ablation study: The frame discriminator improves sharpness, while the sequence discriminator promotes natural expressions and realistic blink distributions.Removing the sequence discriminator drastically reduces blink generation, whereas the full model’s blink statistics resemble real videos.
- Ablation study: The synchronization discriminator contributes to low WER and high audio-visual confidence.The ablation study attributes positive contributions to synchronization and word-recognition performance to the temporal discriminators.
- Qualitative results: Generated videos reproduce accurate lip movements and expressions including frowns, blinks, and angry expressions.Motion maps highlight the regions with the greatest movement, and expressive training data produces a wider expression range.
- Qualitative comparison: Static methods produce jitter and fail to model upper-face motion, whereas the proposed method yields more coherent sequences and more realistic facial movement.Static-model jitter worsens during silent pauses, while Speech2Vid and ATVGNet do not animate the upper face.
- Quantitative results: The proposed method outperforms other approaches in frame quality and content accuracy, with especially substantial differences in lip-reading WER.Its blink amount and duration closely match real videos, although CREMA-D videos show a one-frame lag relative to Speech2Vid.
- Dataset effects: Reconstruction metrics vary by dataset because expressive motion deviates from ground truth and is penalized by reference measures.Audio-visual confidence is lower on the smaller TCD TIMIT and CREMA-D datasets, likely because they provide less articulation data.
7 Conclusion and Future Work
The temporal GAN produces detailed, coherent talking-head videos with accurate mouth movements and spontaneous facial gestures, including blinks, head and brow movements. User testing found the synthesized videos difficult to distinguish from real ones, while the method remains limited to well-aligned frontal faces.
- High PSNR and SSIM, together with strong sharpness across multiple datasets, are mainly attributed to the Frame Discriminator.The ablation study quantifies the Frame Discriminator’s contribution to frame quality.
- The model generates more coherent sequences and more accurate mouth movements than the GAN-based static baseline and Speech2Vid, reflected by a substantial WER difference.The authors associate these improvements with the temporal generator and Synchronization Discriminator.
- Adversarial training over entire sequences enables spontaneous blinks, head movements, brow movements, and facial expressions reflecting the speaker’s emotion.Speech2Vid and ATVGNet are described as unable to generate facial expressions.
- The online Turing test found that users could not distinguish synthesized videos from real ones.The conclusion presents this result as evidence that the generated videos look natural.
- The current method works only for well-aligned frontal faces, with future work targeting in-the-wild conditions and high-definition video.