PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert

Jiadong Wang, Xinyuan Qian, Malu Zhang, Robby T. Tan, Haizhou Li

arXiv:2303.17480v1cs.CVcs.AIeess.IV

TL;DR

Talking-face generation has largely focused on synchronization and visual quality, leaving the intelligibility of spoken content underemphasized. TalkLip uses a lip-reading expert with synchronized contrastive and global-audio modeling, and reports superior intelligibility with competitive synchronization and visual quality. The paper also introduces a dedicated intelligibility evaluation strategy.

  • Problem

    Existing talking-face work emphasizes lip-speech synchronization and visual quality but does not explicitly ensure that generated lip movements convey the correct spoken text.

  • Method

    TalkLip uses a pretrained lip-reading expert to penalize incorrect generated lip movements, adds expert-assisted contrastive synchronization learning, and encodes entire utterances with a global transformer.

  • Results

    TalkLip is superior in reading intelligibility and achieves comparable or better visual quality and lip-speech synchronization than other state-of-the-art methods, including over 38% WER on LRS and 27.8% accuracy on LRW.

  • Takeaways & Limitations

    Reading intelligibility can be directly supervised and benchmarked in talking-face generation through lip-reading-based training and evaluation.

  • Takeaways & Limitations

    The proposed losses and global audio encoder do not improve visual quality, with TalkLip variants showing similar PSNR and SSIM.

Abstract

from arXiv · show

Talking face generation, also known as speech-to-lip generation, reconstructs facial motions concerning lips given coherent speech input. The previous studies revealed the importance of lip-speech synchronization and visual quality. Despite much progress, they hardly focus on the content of lip movements i.e., the visual intelligibility of the spoken words, which is an important aspect of generation quality. To address the problem, we propose using a lip-reading expert to improve the intelligibility of the generated lip regions by penalizing the incorrect generation results. Moreover, to compensate for data scarcity, we train the lip-reading expert in an audio-visual self-supervised manner. With a lip-reading expert, we propose a novel contrastive learning to enhance lip-speech synchronization, and a transformer to encode audio synchronically with video, while considering global temporal dependency of audio. For evaluation, we propose a new strategy with two different lip-reading experts to measure intelligibility of the generated videos. Rigorous experiments show that our proposal is superior to other State-of-the-art (SOTA) methods, such as Wav2Lip, in reading intelligibility i.e., over 38% Word Error Rate (WER) on LRS2 dataset and 27.8% accuracy on LRW dataset. We also achieve the SOTA performance in lip-speech synchronization and comparable performances in visual quality.

1. Introduction

Talking face generation has emphasized synchronization and visual quality, while reading intelligibility—the recognizability of spoken content from lip movements—has received less attention. TalkLip addresses this gap with a lip-reading expert, synchronized audio-visual learning, and a new intelligibility evaluation strategy.

  • Motivation: Reading intelligibility measures how much spoken text humans can interpret from generated face videos, especially for hearing-impaired users.Image quality and lip-speech synchronization do not explicitly measure whether the generated lip movements convey the correct text.
  • Motivation: Synchronized or high-quality videos can still convey incorrect phonetic content, motivating explicit evaluation and supervision of lip movements.The McGurk effect illustrates that unpaired audio and lip movements can produce phonetic or fused percepts.
  • Approach: TalkLip uses a lip-reading expert to transcribe generated image sequences and penalize incorrect lip-reading predictions during generator training.Generated frames are substituted into face sequences and passed to the expert, whose supervision reaches the face generator.
  • Approach: TalkLip introduces expert-assisted cross-modal contrastive learning to enhance lip-speech synchronization.The method contrasts audio embeddings used for generation with visual context features from the lip-reading encoder.
  • Approach: A synchronously trained transformer audio encoder models global temporal dependency across the entire speech utterance.This complements short-term local audio representations based on 0.2-second segments.
  • Evaluation: 38% WER on LRS and 27.8% accuracy on LRW are reported for reading intelligibility, with comparable or better visual quality and synchronization than prevailing methods.The evaluation uses a new intelligibility strategy and reports superiority over other methods on reading intelligibility.

2. Related work

Related talking-face methods mainly use intermediate facial representations or end-to-end reconstruction, while TalkLip adds lip-reading supervision to either category. Its synchronization strategy uses dynamic lip features and contrastive learning rather than single-image visual features.

  • Talking-face generation: Traditional talking-face generation mapped acoustic features to time-aligned facial motions, whereas recent deep methods use intermediate representations or reconstruction.Intermediate methods predict landmarks or 3D meshes before synthesis; reconstruction methods generate videos end to end with encoder-decoder architectures.
  • Intermediate representation-based methods: Intermediate representation methods predict speech- and identity-related landmarks or 3D facial meshes before generating face videos.Examples include landmark prediction from speech and disentanglement of speech content and speaker identity.
  • Reconstruction-based methods: Reconstruction-based methods use end-to-end architectures to improve synchronization, visual representations, or speech representations.Prior work applies synchronization discriminators, associative and adversarial training, or disentangles phonetic and emotional speech content.
  • TalkLip positioning: TalkLip’s lip-reading-expert penalty can extend to both intermediate-representation and reconstruction-based generation, although the paper demonstrates it with reconstruction.The proposed system combines speech, identity, and pose inputs with generated-face supervision from the expert.
  • Contrastive learning: Contrastive learning aligns same-class features while separating different-class features in a shared representation space.This framework can support self-supervised learning without labeled data.
  • Contrastive learning for synchronization: Unlike PC-AVS, TalkLip extracts visual features from five frames to represent dynamic lip movement rather than a single image.PC-AVS uses temporally aligned audio and visual lip features as positive pairs and others as negatives.

3. Proposed Methods

TalkLip combines a frozen lip-reading expert, hybrid-video supervision, synchronized audio encoding, and generation losses to improve intelligible talking-face synthesis.

  • Lip-reading-guided generation: The lip-reading expert transcribes generated face sequences and penalizes incorrectly generated lip movements during generator training.It is applied to hybrid videos containing ground-truth frames with selected synthesized frames, allowing gradients to optimize the generator and encoders.
  • Lip-reading-guided generation: AV-Hubert provides a self-supervised lip-reading expert whose shared pseudo-label training synchronizes audio and visual context features.The pretrained visual frontend and transformer encoder are fine-tuned with a text decoder, then frozen for generation training.
  • Synchronized audio and visual encoding: The audio encoder supplies phoneme-level embeddings, using local 0.2-second features and global embeddings from a transformer processing the entire utterance.The global embedding is selected at the frame aligned with the pose reference, capturing longer-term temporal dependency.
  • Synchronized audio and visual encoding: The video encoder combines identity and pose references into a visual embedding for identity- and pose-consistent image synthesis.The pose reference masks the lower face, while the identity reference is sampled from the same video sequence.
  • Video generation: The generator uses a U-Net-style skip connection and transposed convolutional blocks to synthesize talking-face images.The skip connection balances reconstruction quality and computational efficiency.
  • Video generation: Reconstruction and GAN losses improve generation quality, while the GAN discriminator distinguishes synthesized images from ground-truth faces.The total objective combines lip-reading, contrastive, reconstruction, and GAN losses with separate scaling coefficients.
  • Synchronization learning: Contrastive learning attracts time-aligned audio and visual context embeddings while repelling audio embeddings from different frames.InfoNCE is computed using audio embeddings and frozen lip-reading visual features from masked frames.

4. Experiments

Experiments evaluate TalkLip on visual quality, lip-speech synchronization, and reading intelligibility across LRS2 and unseen LRW, including quantitative, qualitative, user-study, and ablation analyses. TalkLip improves reading intelligibility and synchronization, while visual quality remains competitive under limited training data.

  • Experimental settings: Experiments train on 29 hours of LRS2 data and evaluate against five SOTA methods on LRS2 and LRW.The evaluation includes visual quality, synchronization, and reading intelligibility metrics.
  • Quantitative results: TalkLip achieves SOTA lip-speech synchronization with global audio encoding and contrastive learning, although Wav2Lip has higher LSE-C than some TalkLip variants.The comparison also shows that stronger synchronization does not necessarily produce better reading intelligibility.
  • Quantitative results: TalkLip methods outperform competing methods in reading intelligibility on LRS2 and unseen LRW, reaching performance close to or better than real videos.The advantage spans continuous-utterance and single-word lip-reading tasks.
  • Qualitative and user-study results: TalkLip produces competitive visual quality and the best subjective results, scoring 3.76 in reading intelligibility and exceeding Wav2Lip by 0.38.Its visual quality is below SyncTalkFace, which uses substantially more training data, but remains close to Wav2Lip.
  • Ablation study on the GAN loss: GAN loss improves lip-speech synchronization and reading intelligibility in addition to making synthesized videos more realistic.Removing the GAN loss lowers LSE-C by about 19.5% on LRW and LRS2 and particularly reduces reading intelligibility on LRW.
  • Ablation study on the lip-reading loss: 26.5% improvement on LRS2 and 31.2% improvement on LRW result from adding lip-reading loss to TalkLip (l).These ablations support the contribution of the lip-reading expert to generated-face intelligibility.
  • Ablation study on contrastive learning: At least 6.5% WER improvement on LRS2 and 3.39% ACC improvement on LRW result from adding contrastive learning to TalkLip (l).The corresponding t-SNE visualization shows ‘Meeting’ and ‘Media’ embeddings mixed without contrastive learning but only three outliers with it.
  • Global audio encoding: The global audio encoder improves both reading intelligibility and lip-speech synchronization over the local encoder by using entire-utterance temporal information.Its synchronically pretrained audio representations provide phoneme-level information aligned with visual content features.

5. Conclusion

The paper presents TalkLip, which targets reading intelligibility in talking-face generation with a lip-reading expert. It additionally uses contrastive learning and a synchronically pretrained audio transformer, and experiments report superior reading intelligibility, effective synchronization improvements, and competitive visual quality.

  • Conclusion: TalkLip uses a pretrained lip-reading expert to penalize incorrect lip-reading predictions and improve the intelligibility of synthesized talking faces.The evaluation introduces word accuracy for single words and WER for continuous utterances.
  • Conclusion: Contrastive learning improves lip-speech synchronization and also improves reading intelligibility.The method uses the lip-reading expert to support this contrastive objective.
  • Conclusion: A synchronically pretrained audio transformer leverages global temporal dependency to generate better phoneme-level audio embeddings and support synchronization.The conclusion attributes this benefit to the transformer's synchronized pretraining and global temporal modeling.
  • Conclusion: TalkLip nets outperform competitive methods in reading intelligibility by a large margin while achieving effective synchronization and competitive visual quality.The conclusion summarizes evidence from extensive experiments across the proposed evaluation measures.

6. Supplementary

The supplementary section specifies local and global audio encoders. Local embeddings are extracted from 0.2-second spectrogram segments, while the transformer captures phoneme-level information with global temporal dependency.

  • Audio Encoder: The local audio encoder extracts embeddings from spectrograms representing 0.2-second audio segments.Its convolutional blocks use residual connections, and batch normalization follows each convolutional layer.
  • Audio Encoder: The transformer encoder processes variable-length speech to capture phoneme-level information and global temporal dependency.Speech is represented as a spectrogram, with every four frames stacked and 26 filter banks fixed.
  • Audio Encoder: The transformer contains 12 cascaded blocks with hidden dimension 768, feed-forward dimension 3072, and 12 attention heads.Its output has shape R^(T/4)×768, and a frame aligned with the pose reference becomes the global audio embedding.

6.3. Video Encoder Architecture

The video and generator architectures combine identity, pose, and audio information to synthesize face images. Skip connections transfer matching hidden features from the video encoder into the generator.

  • Video Encoder Architecture: The video encoder combines an identity image and a pose image into a unified visual embedding containing identity and pose information.The concatenated input has shape 6×96×96.
  • Generator Architecture: The generator synthesizes a face image from concatenated audio and video embeddings.Its detailed architecture is specified in Table 5.
  • Generator Architecture: Unet-like skip connections concatenate generator features with same-shaped hidden features from the video encoder.This connects corresponding feature representations across the two networks.

6.5. Discriminator Architecture

The discriminator evaluates the lower half of synthesized faces, while qualitative analyses show that contrastive learning improves synchronization and embedding separation for frequently confused words.

  • Discriminator Architecture: The discriminator penalizes unrealistic synthesized face images using only the lower half of each face as input.Its architecture is described in Table 6.
  • Qualitative Analysis: Contrastive learning improves lip-speech synchronization and reading intelligibility for the frequently confused words ‘Around’ and ‘Ground’.The study visualizes their audio embeddings to examine this effect.
  • Qualitative Analysis: TalkLip (l + c) produces more separated word-related embeddings than TalkLip (l) and shows better synchronization with the ground truth.The comparison is illustrated through embedding visualization and generated-video snapshots.

6.7. Qualitative Ablation Study on Global Audio Embedding

Global audio embeddings provide better phoneme-level separation than local embeddings and produce fuller lip movements. LSE-D comparisons further confirm TalkLip (g + c)’s state-of-the-art synchronization performance.

  • Global Audio Embedding: Global audio embeddings separate ‘Around’ and ‘Ground’ more clearly than local audio embeddings.This visualization supports the benefit of representing phoneme-level information with global temporal context.
  • Global Audio Embedding: TalkLip (g + c) produces fuller lip movements than TalkLip (l + c).The comparison is presented as evidence for the benefit of global audio embeddings.
  • Synchronization Evaluation: LSE-D confirms TalkLip (g + c)’s state-of-the-art performance on lip-speech synchronization.The comparison appears in Table 7.

6.9. Limitation

The reported methods do not improve visual quality, as TalkLip variants show similar PSNR and SSIM. The authors therefore leave visual-quality improvement for future work.

  • 6.9. Limitation: TalkLip methods do not improve visual quality, with similar PSNR and SSIM across variants.The lip-reading loss, contrastive learning, and global audio encoder do not boost these visual-quality measures.
  • 6.9. Limitation: The authors identify improving visual quality as future work.

6.10. Qualitative Result

The paper presents additional qualitative comparisons against ATVGnet, Wav2Lip, and Faceformer. These comparisons are provided in Figures 9–12 to demonstrate the proposal's superiority.

  • 6.10. Qualitative Result: The qualitative evaluation compares the proposal with ATVGnet, Wav2Lip, and Faceformer.The authors identify these as three state-of-the-art methods used for comparison.
  • 6.10. Qualitative Result: Figures 9–12 contain the additional qualitative comparisons.
  • 6.10. Qualitative Result: The qualitative comparisons are intended to show the superiority of the proposal.
Loading 2303.17480v1…