Source-linked AI summary
Talking Face Generation by Conditional Recurrent Adversarial Network
Yang Song, Jingwen Zhu, Dawei Li, Xiaolong Wang, Hairong Qi
TL;DR
The paper addresses talking-face generation from an arbitrary face image and speech clip, where accurate lip synchronization, temporal smoothness, realism, and identity preservation remain challenging. It proposes a conditional recurrent adversarial framework with image-audio temporal modeling and specialized discriminators, and reports superiority over state-of-the-art methods across controlled and uncontrolled data. The authors also identify raw-audio end-to-end modeling and sentence-level lip-reading guidance as directions for further investigation.
Problem
Generating realistic talking-face videos from one face image and speech requires identity preservation, accurate lip shapes, and photo- and video-realism despite temporal and generalization challenges.
Method
The framework recurrently conditions generation on audio and image information and uses spatial-temporal and lip-reading discriminators for realism and lip-motion accuracy.
Results
The framework demonstrates superiority over state-of-the-art methods under different performance metrics on public constrained and unconstrained data.
Takeaways & Limitations
The approach supports talking-face generation with smoother facial and lip transitions while improving visual realism and lip-synchronization accuracy.
Takeaways & Limitations
The framework uses MFCC features rather than raw audio, and the authors propose true end-to-end raw-audio generation and sentence-level lip-reading guidance as further work.
Abstract
from arXiv · showhide
Given an arbitrary face image and an arbitrary speech clip, the proposed work attempts to generating the talking face video with accurate lip synchronization while maintaining smooth transition of both lip and facial movement over the entire video clip. Existing works either do not consider temporal dependency on face images across different video frames thus easily yielding noticeable/abrupt facial and lip movement or are only limited to the generation of talking face video for a specific person thus lacking generalization capacity. We propose a novel conditional video generation network where the audio input is treated as a condition for the recurrent adversarial network such that temporal dependency is incorporated to realize smooth transition for the lip and facial movement. In addition, we deploy a multi-task adversarial training scheme in the context of video generation to improve both photo-realism and the accuracy for lip synchronization. Finally, based on the phoneme distribution information extracted from the audio clip, we develop a sample selection method that effectively reduces the size of the training dataset without sacrificing the quality of the generated video. Extensive experiments on both controlled and uncontrolled datasets demonstrate the superiority of the proposed approach in terms of visual quality, lip sync accuracy, and smooth transition of lip and facial movement, as compared to the state-of-the-art.
1 Introduction
Talking face generation must coordinate identity, lip synchronization, and realistic, temporally smooth facial motion from a still image and speech. The proposed framework addresses these demands with recurrent image-audio conditioning and complementary adversarial discriminators.
- Talking face generation synthesizes a face video from a still facial image and speech, with applications including bandwidth reduction and face animation.
- Video generation is sensitive to temporal discontinuities, while speech conditioning imposes strict lip-synchronization requirements and large pose, speed, and style variations challenge generalization.
- Many prior methods generate frames independently, omitting facial temporal dependency and inadequately modeling coarticulation.
- The proposed recurrent unit incorporates image and audio, including previously generated frames, to model temporal dependency and smooth both lip and facial movements.
- The framework uses spatial-temporal discriminators for image- and video-realism and a lip-reading discriminator to improve lip synchronization.
2 Related Work
Related work spans person-specific lip modeling, end-to-end 3D mesh animation, video prediction, and adversarial video generation. Closely related image-based approaches still face challenges in identity consistency, video realism, or temporal modeling.
- Suwajanakorn et al. trained a lip model for President Obama, but rigid matching and mouth-motion variation make adaptation to other people difficult.
- Karras et al. mapped raw audio to 3D meshes end-to-end, but mesh animation cannot capture tongue, wrinkles, eyes, or head motion essential for image-level face animation.
- Video prediction and adversarial generation methods model future frames or motion and background content, but some are restricted to fixed-length videos or disentangled generation settings.
- A lip-region method faces identity and video-realism challenges, while a whole-face method stacks independently generated frames and therefore lacks modeled temporal dependency.
3 Proposed Method
The proposed method generates talking-face videos from an audio sequence and one identity image while modeling long-term visual and audio dependencies. It combines recurrent generation with image, video, and lip-reading adversarial objectives to preserve identity, improve realism, and synchronize lip movements.
- 3.1 Problem Formulation: The task generates a frame sequence conditioned on an audio sequence and a single identity image.The generated conditional distribution is intended to approximate the real talking-face video distribution.
- 3.1 Problem Formulation: Frame-to-frame generation ignores temporal dependence, while sequential generation models only short-term dependence and can lose identity over long durations.The recurrent scheme is introduced to address these limitations.
- 3.2 Conditional Recurrent Video Generation: The hybrid recurrent scheme incorporates image and audio features into a recurrent unit to model visual and audio dependencies over time.The network ingests sequential image and audio signals and generates the image sequence through a decoder.
- 3.2 Conditional Recurrent Video Generation: Audio segments are represented with MFCC features, while a randomly selected identity image is encoded and combined with audio features before decoding target frames.The middle frame aligned with each audio segment supplies the corresponding target lip shape.
- 3.3 Adversarial Learning: A lip-reading discriminator provides semantic guidance for lip synchronization because reconstruction loss alone uses averaged pixel-wise distance.Its training pushes predictions for real sequences toward true labels while forcing fake-sequence predictions to be misclassified.
- 3.3 Adversarial Learning: Image and video discriminators jointly improve realism, with the video discriminator mainly promoting smoothness and continuity between generated frames.The image discriminator focuses relatively more on individual-frame enhancement, whereas the video discriminator targets temporal transitions.
4 Experimental Results
Experiments across three datasets evaluate image quality, lip accuracy, and video smoothness. The proposed recurrent framework outperforms compared methods qualitatively and quantitatively, while ablations associate its components with sharper details, clearer lip movement, and smoother temporal behavior.
- Experimental Setup: Experiments use TCD-TIMIT, LRW, and VoxCeleb, covering controlled speech data, short real-world clips, and videos with substantial variation.The evaluation includes synchronized face and audio inputs, with 350 ms MFCC windows and mouth-only inputs for the lip-reading discriminator.
- Qualitative Evaluation: The proposed method produces sharper skin texture, more realistic wrinkles, clearer teeth, and more discriminative mouth shapes than the compared methods.The authors attribute visual-detail improvements to spatial-temporal discrimination and sharper mouth shapes partly to the lip-reading discriminator.
- Qualitative Evaluation: Recurrent generation reduces inter-frame discontinuities and motion inconsistency by modeling temporal dynamics across frames.Compared with frame-to-frame generation, it preserves identity and concentrates most optical-flow movement around the mouth; it also helps avoid pose inconsistency.
- Qualitative Evaluation: Adding image adversarial loss improves detail, video adversarial loss further sharpens images and reduces jitter, and lip-reading loss makes lip movement more obvious.The recurrent network contributes especially to global video smoothness, including avoiding zoom-in and zoom-out effects.
- User Study: User studies report that the proposed method outperforms recent approaches in lip movement accuracy, video realism, and image quality.The compared methods exhibit motion artifacts including pose discontinuity and unstable face sizes between frames.
5 Study on Natural Pose and Expression
The modified network targets natural pose and expression by incorporating previously generated frames alongside hybrid features. On the Obama dataset, it produces accurate lip shapes and natural pose changes.
- 5 Study on Natural Pose and Expression: The 2D approach addresses natural pose and expression by feeding the previously generated image frame into the next recurrent unit.This combines sequential generation with recurrent generation so pose and expression can be modeled intrinsically.
- 5 Study on Natural Pose and Expression: Training the modified network on the Obama dataset produced videos with accurate lip shapes and natural pose changes.
- 5 Study on Natural Pose and Expression: Figure 6 compares ground-truth images in its first row with the method’s generated results in its second row.
6 Conclusion, Limitations and Future Work
The paper concludes that its conditional adversarial network improves talking-face generation across constrained and unconstrained data. It identifies raw-audio end-to-end modeling, sentence-level lip reading, and super-resolution as future directions.
- 6 Conclusion: The conditional adversarial network captures temporal dependencies in audio and image information while using three discriminators for image quality, video realism, and lip-movement accuracy.
- 6 Conclusion: Experiments on public constrained and unconstrained datasets demonstrated superiority over state-of-the-art methods under different performance metrics.
- 6 Limitations and Future Work: Future work includes an end-to-end framework that takes raw audio instead of MFCC features as network input.
- 6 Limitations and Future Work: Further directions include sentence-level lip-reading guidance and incorporating super-resolution to improve image quality.