Source-linked AI summary
Lip Movements Generation at a Glance
Lele Chen, Zhiheng Li, Ross K. Maddox, Zhiyao Duan, Chenliang Xu
TL;DR
The paper addresses generating a target identity’s lip movements from arbitrary speech and one lip image, requiring realistic identity preservation and synchronized temporal motion. It fuses audio and image embeddings, generates multiple frames jointly, and combines complementary losses. On three datasets, the model significantly improves over state-of-the-art methods extended to this task, with scope boundaries around fixed-length output and single-image modeling of person-specific deformation.
Problem
The task is to generate realistic, smooth, identity-preserving lip-motion video from arbitrary speech and one target-identity lip image while modeling audio-lip correlations.
Method
The method fuses speech and identity-image embeddings to generate 16 frames jointly and combines correlation, adversarial, perceptual, and reconstruction losses end to end.
Results
Significant improvement over two state-of-the-art methods is reported on GRID, LDC, and LRW.
Takeaways & Limitations
Jointly modeling speech-lip correlations and multiple-frame generation improves this lip-movement generation task across varied datasets.
Takeaways & Limitations
The authors identify fixed-length generation and difficulty capturing person-specific lip deformation from a single image as limitations or future boundaries.
Abstract
from arXiv · showhide
Cross-modality generation is an emerging topic that aims to synthesize data in one modality based on information in a different modality. In this paper, we consider a task of such: given an arbitrary audio speech and one lip image of arbitrary target identity, generate synthesized lip movements of the target identity saying the speech. To perform well in this task, it inevitably requires a model to not only consider the retention of target identity, photo-realistic of synthesized images, consistency and smoothness of lip images in a sequence, but more importantly, learn the correlations between audio speech and lip movements. To solve the collective problems, we explore the best modeling of the audio-visual correlations in building and training a lip-movement generator network. Specifically, we devise a method to fuse audio and image embeddings to generate multiple lip images at once and propose a novel correlation loss to synchronize lip changes and speech changes. Our final model utilizes a combination of four losses for a comprehensive consideration in generating lip movements; it is trained in an end-to-end fashion and is robust to lip shapes, view angles and different facial characteristics. Thoughtful experiments on three datasets ranging from lab-recorded to lips in-the-wild show that our model significantly outperforms other state-of-the-art methods extended to this task.
1 Introduction
The paper introduces lip-movement generation from arbitrary speech and a single target-identity lip image, addressing identity preservation, realism, temporal smoothness, and audio-visual correspondence. It generates multiple frames jointly and reports significant improvement over state-of-the-art extensions on three datasets.
- Task and motivation: The task generates a target identity’s lip movements from arbitrary speech and one lip image, without requiring the speech or target image during training.The setting separates the speaker in the audio from the target identity in the image.
- Task and motivation: Existing work mainly generates talking faces for a target identity, while the closest prior method generates one image per audio segment and bypasses sequence consistency and temporal correlations.The paper identifies consistency, smoothness, and audio-lip temporal relationships as unresolved challenges.
- Method overview: The proposed method fuses time-series audio and identity-image embeddings to generate 16 lip images jointly in a video.Joint generation is intended to depict the corresponding lip movements of the target identity.
- Method overview: The final model combines audio-visual correlation, three-stream adversarial, feature-space, and pixel-level reconstruction losses in an end-to-end training framework.The design targets synchronization, image and motion quality, perceptual similarity, and pixel-level fidelity.
- Evaluation: The model is evaluated on GRID, LDC, and LRW using landmark distance plus PSNR, SSIM, and CPBD image-quality metrics.The datasets range from lab-recorded speech to lip movements in the wild.
- Evaluation: The final model achieves significant improvement over state-of-the-art methods extended to this task across the three datasets.The paper presents this cross-dataset comparison as a principal contribution.
2 Related Work
Related work spans cross-modal generation, lip reading, audiovisual synchronization, and adversarial training. The paper distinguishes its task from prior lip-generation systems that do not jointly address sequence consistency and temporal audio-lip correlations.
- Cross-modal generation: Cross-modal research includes video-to-sound, text-to-image, and sound-to-image generation alongside established image/video-to-text tasks.These examples frame lip-movement generation as a cross-modality problem.
- Lip reading: Lip reading uses correlations between lip movements and spoken sentences or words to interpret audio information from visual input.Related approaches learn shared or correlated audiovisual representations using CCA or temporal neural networks.
- Audiovisual synchronization: Human lips can move before speech audio, so audiovisual models must account for temporal delay rather than assume perfect alignment.Prior time-delayed RNNs use a manually fixed delay, which is difficult to determine for in-the-wild videos.
- Adversarial training: Adversarial training can condition generation on additional information and help bridge modalities, but the paper reports better results when GAN loss is combined with other losses.This motivates using adversarial learning as one component of a broader objective.
3 Lip-Movement Generator Network
The generator encodes speech and target identity separately, transforms them into a shared spatiotemporal representation, and decodes multiple lip frames. Its correlation networks compare temporal audio and visual changes while accommodating audiovisual delay.
- Input encoding: The network encodes log-mel speech spectrograms into audio features and a target identity image into visual features.Audio features are organized over time and frequency, while visual features represent image height and width.
- Audio-identity fusion: Audio and visual features are fused into a synthesized video feature, which residual blocks and 3D deconvolutions expand into a video.The fusion must preserve target identity while capturing speech-driven temporal variation.
- Audio-identity fusion: The fusion duplicates audio features across frequency and image features across time, then concatenates them along the channel dimension.The method sets the spatial feature dimensions H and W equal to the audio feature dimension F.
- Derivative correlation: The correlation loss compares encoded consecutive audio changes with encoded optical-flow changes using cosine similarity.Audio derivatives represent speech changes, while optical flow represents visual lip-motion changes.
- Derivative correlation: A study of 3,260 GRID videos finds that different videos favor different offsets for maximum audio-flow correlation.The result argues against imposing one fixed audiovisual delay across videos.
- Delay modeling: Correlation networks use 3D CNNs with large temporal receptive fields to match feature sizes and learn delay from data rather than fixing it as a hyperparameter.The networks retain temporal length while reducing feature size for correlation computation.
4 Full Model and Training
The full model is trained end-to-end to generate realistic lip-movement videos from paired speech and identity images. It combines four losses and uses a three-stream discriminator to assess appearance, motion, and audio-visual matching.
- Training Setup: Training pairs each lip-movement video with its corresponding speech audio and samples an identity frame from the same source video.At test time, speech and the identity image may come from outside the training dataset.
- Loss Design: The objective combines correlation, pixel reconstruction, perceptual, and adversarial losses.The coefficients for the pixel, perceptual, and adversarial terms are set to 0.5, 1.0, and 1.0, respectively.
- Loss Design: Correlation loss aligns audio and visual information, while pixel reconstruction retains identity texture and perceptual loss improves synthesized-frame sharpness.Pixel loss alone is reported to reduce frame sharpness, motivating the additional perceptual term.
- Three-Stream GAN Discriminator: The three-stream discriminator conditions on speech while judging video appearance and explicitly modeling motion through an optical-flow stream.Its streams process audio, video, and optical flow before their features are concatenated for discrimination.
- Three-Stream GAN Discriminator: The discriminator also uses mismatched audio-video pairs so that it becomes sensitive to audio-visual correspondence.The mismatch strategy includes a real video paired with noncorresponding audio or video.
5 Experiments
Experiments evaluate the model across three datasets using landmark-based lip-motion accuracy and image-quality metrics, plus ablations, state-of-the-art comparisons, and qualitative tests. The results indicate improved accuracy and visual quality, while revealing limitations in identity, skin-color, and deformation preservation.
- Datasets and Settings: Experiments use GRID, LDC, and LRW, spanning lab-recorded videos and in-the-wild news footage.Testing sets contain unseen speakers and unseen sentences.
- Evaluation Metrics: LMD measures normalized Euclidean distance between corresponding detected lip landmarks, while SSIM, PSNR, and CPBD assess image similarity and sharpness.LMD uses temporal length and 20 landmark points for normalization.
- Ablation Study: Removing correlation loss worsens LMD, SSIM, and PSNR, while derivative-based correlation outperforms a non-derivative variant on these metrics.The non-derivative variant uses direct audio features and video frames instead of audio derivatives and optical flow.
- Ablation Study: The three-stream discriminator improves results over a two-stream discriminator, and optical flow outperforms frame differences for modeling motion changes.The full model produces higher CPBD and lower LMD than the frame-difference variant.
- Comparison to State-of-the-Art: The full model outperforms extended state-of-the-art methods on most metrics across three datasets, including LMD and PSNR.Reconstruction-based comparisons produce blurrier images, reflected in the CPBD results.
- Qualitative and Real-World Examples: Qualitative tests show sharper frames and robustness to view angles, lip shapes, and facial characteristics, but identity and skin-color preservation can fail.The model also struggles to capture person-specific lip deformation from a single image.
6 Conclusion and Future Work
The paper addresses identity-conditioned lip-movement generation from arbitrary speech and a single target-identity lip image by combining a new generator, audio-visual correlation loss, and four complementary losses. It reports significant improvements on three datasets and identifies variable-length generation and full-face generation as future directions.
- The task generates synthesized lip movements for an arbitrary target identity from arbitrary speech audio and one lip image.
- The approach combines a new generator network, an audio-visual correlation loss, and four complementary losses.
- The model shows significant improvements over two state-of-the-art methods on three datasets.
- Future Work: Future work includes non-fixed-length lip-movement generation and end-to-end full-face generation.
Appendix A Network Architectures
The appendix documents the architectures used throughout the system, covering separate encoders, a decoder, and a three-stream GAN discriminator. The listed components include audio, identity, audio-derivative, flow, video, and optical-flow processing streams.
- Encoders and Decoder: The appendix specifies separate Audio Encoder, Identity Encoder, Decoder, Audio Derivative Encoder, and Flow Encoder architectures.
- Three-stream GAN Discriminator: The Three-stream GAN discriminator contains separate audio, video, and optical-flow streams.
- Architectural Conventions: The architecture tables define Conv as Convolution, Batch Normalization, and ReLU, and Trans. Convs as Transposed Convolution, Batch Normalization, and ReLU.
- Encoders and Decoder: The Audio Derivative Encoder is denoted as φs, while the Flow Encoder is denoted as φv in the main paper.