Source-linked AI summary
Talking Face Generation by Adversarially Disentangled Audio-Visual Representation
Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, Xiaogang Wang
TL;DR
Talking face generation must preserve subject identity while conveying speech, but identity and speech signals are coupled in facial motion. The paper learns disentangled audio-visual representations with associative and adversarial training, producing arbitrary-subject talking faces from audio or video and supporting lip reading and retrieval.
Problem
Talking face generation must preserve identity while clearly conveying speech, despite subject-related and speech-related information being coupled in facial deformation.
Method
DAVS learns a joint audio-visual speech representation, disentangles it from person identity through adversarial learning, and combines both spaces for generation.
Results
DAVS generates high-quality, temporally accurate talking faces for arbitrary subjects using either audio or video speech inputs, while improving lip reading and supporting audio-video retrieval.
Takeaways & Limitations
The learned disentangled representation unifies audio-visual speech recognition, synchronization, and arbitrary-subject talking face generation in an end-to-end framework.
Abstract
from arXiv · showhide
Talking face generation aims to synthesize a sequence of face images that correspond to a clip of speech. This is a challenging task because face appearance variation and semantics of speech are coupled together in the subtle movements of the talking face regions. Existing works either construct specific face appearance model on specific subjects or model the transformation between lip motion and speech. In this work, we integrate both aspects and enable arbitrary-subject talking face generation by learning disentangled audio-visual representation. We find that the talking face sequence is actually a composition of both subject-related information and speech-related information. These two spaces are then explicitly disentangled through a novel associative-and-adversarial training process. This disentangled representation has an advantage where both audio and video can serve as inputs for generation. Extensive experiments show that the proposed approach generates realistic talking face sequences on arbitrary subjects with much clearer lip motion patterns than previous work. We also demonstrate the learned audio-visual representation is extremely useful for the tasks of automatic lip reading and audio-video retrieval.
1 Introduction
The paper targets arbitrary-subject talking face generation from a single face image and speech represented by either video or audio. It addresses the coupling of identity and speech information by learning disentangled audio-visual representations.
- Arbitrary-subject talking face generation seeks speech-synchronized videos from one target face image and either video or audio speech input.
- Talking-face deformation reflects both intrinsic subject traits and extrinsic speech vibrations, making identity-preserving generation with clear speech communication difficult.
- The proposed approach separates talking-face sequences into complementary identity and speech representations despite large variation across subjects and speeches.
- Audio-visual speech recognition associates video and audio representations by aligning lip-reading results with speech-recognition results, followed by adversarial disentanglement.
- The framework learns a joint representation, disentangles person identity from speech, and supports end-to-end generation from either video or audio guidance.
2 Related Work
Earlier work combines subject-specific face models, audio-to-lip-motion synthesis, or audio-visual representation learning. These approaches motivate a framework that connects speech representations with arbitrary-identity face generation.
- Generating Talking Faces: Computer-graphics methods animate subject-specific 3D face models, but their reliance on chosen identities makes scaling to arbitrary subjects difficult.
- Generating Talking Faces: Deep generative approaches support arbitrary subjects, while autoencoder-based audio-to-face generation can produce blurry results and lack continuity.
- Learning Audio-Visual Representation: Audio-visual speech recognition uses video, audio, or both for word- or sentence-level recognition, supported by CNNs, RNNs, and large-scale datasets.
- Learning Audio-Visual Representation: Audio-video synchronization methods align mouth motion with speech, including SyncNet’s two-stream CNN alignment of audio MFCCs with five consecutive frames.
- The proposed framework separates person identity and speech information into pid and wid spaces, using encoders, a decoder, and adversarial components.
3 Approach
DAVS learns joint audio-visual and person-identity representations, explicitly disentangles speech and subject information, and combines them to generate talking faces from video or audio.
- DAVS is an end-to-end trainable network for talking face generation using disentangled audio-visual representations.
- The system learns person-identity and Word-ID spaces from talking video and corresponding audio using dedicated encoders.The Word-ID space captures speech-related information, while the Person-ID space captures subject-related information.
- 3.1 Learning Joint Audio-Visual Representation: Shared classification, contrastive, and domain-adversarial objectives associate paired audio-video features and make their distributions indistinguishable.The shared classifier aligns modality features toward class centroids, contrastive loss brings paired samples closer, and domain adversarial training prevents source discrimination.
- 3.2 Adversarial Training for Latent Space Disentangling: Adversarial training removes Person-ID information from the Word-ID representation and Word-ID information from the Person-ID representation.The encoder is updated against a fixed classifier so the undesired information becomes uninformative to the discriminator.
- 3.3 Inference: Arbitrary-Subject Talking Face Generation: Generation combines a Person-ID feature from a random guidance frame with either video or audio Word-ID features through decoder G.The guidance feature supplies identity, while the Word-ID feature supplies speech information.
- 3.3 Inference: Arbitrary-Subject Talking Face Generation: The reconstruction objective combines L1 reconstruction and temporal GAN losses, with the latter used to improve sharpness and continuity.The combined loss is written as LRe, with α weighting the two terms.
4 Experiments
Experiments evaluate arbitrary-subject generation, audio-visual representation learning, and identity-speech disentanglement across reconstruction, user-study, recognition, retrieval, and ablation settings.
- Arbitrary-Subject Talking Face Generation: The model generates talking faces from either audio or video identity guidance, with qualitative comparisons reporting improved identity preservation and image quality over Chung et al.The input identity guidance is any person’s face image, while speech information comes from one audio or video source.
- Arbitrary-Subject Talking Face Generation: User ratings on an unseen dataset favored the proposed generation over the reproduced baseline for synchronization and realism.Ten participants evaluated results averaged across people and time steps without ground-truth videos.
- Audio-Visual Representation: Adding contrastive loss, shared classification, and domain-adversarial training progressively improves audio-visual speech recognition and audio-video retrieval, with all supervisions achieving the highest results.Retrieval uses 25,000 LRW test samples and reports R@1, R@10, and Med R.
- Audio-Visual Representation: Different supervisions change mouth-opening duration and strength; shared classification brings motion closer to ground truth, while adversarial training improves image quality.The same audio clip can produce different lip motions across subjects, so audio-to-lip motion is not one-to-one.
- Identity-Speech Disentanglement: After adversarial disentanglement, person-identity features retain less word-identity information, with correct word assignment decreasing from 27.8% to 9.7%.The experiment uses an SVM to test residual word-identity information in the person-identity encoder.
- Identity-Speech Disentanglement: Adversarial disentanglement improves lip-sync quality, evaluated through identity similarity and landmark-based lip deviation measures.OpenFace squared L2 similarity measures identity distance, while 20 lip landmarks characterize deviation from ground truth.
5 Conclusion
DAVS learns disentangled audio-visual representations to generate high-quality talking-face videos. It supports arbitrary subjects and accepts either audio or video speech information as input guidance.
- DAVS learns a joint audio-visual embedding with discriminative speech information, then separates it from person identity through adversarial learning.
- The framework learns joint audio-visual representations by associating multiple supervisions through audio-visual speech discrimination.
- DAVS unifies audio-visual speech recognition and audio-visual synchronization in an end-to-end framework.
- DAVS enables high-quality, temporally accurate talking-face generation for arbitrary subjects.
- Both audio and video speech information can provide input guidance for talking-face generation.