Source-linked AI summary
You said that?
Joon Son Chung, Amir Jamaludin, Andrew Zisserman
TL;DR
Talking-face generation requires synchronizing speech with a target identity from limited visual input. The paper introduces Speech2Vid, a CNN model that jointly embeds face images and audio, then generates lip-synced frames in real time from unlabelled videos. It demonstrates generation for unseen inputs and video re-dubbing, while noting that quantitative evaluation remains unresolved.
Problem
The paper considers transforming audio into video of a specified target face, including when the speech was not originally spoken by that person.
Method
Speech2Vid uses audio and identity-image encoders with a talking-face decoder and deblurring module, trained unsupervised on tens of hours of talking videos.
Results
The model generates natural-looking talking faces for identities and audio outside the training data, supports multiple-image identity inputs, and enables re-dubbing from an independent audio source.
Takeaways & Limitations
The results support generating talking-face video directly from audio and still images, including re-dubbing videos with speech from another person.
Takeaways & Limitations
The paper does not provide a straightforward quantitative performance measure because generative talking-face quality lacks a definitive domain-specific metric.
Abstract
from arXiv · showhide
We present a method for generating a video of a talking face. The method takes as inputs: (i) still images of the target face, and (ii) an audio speech segment; and outputs a video of the target face lip synched with the audio. The method runs in real time and is applicable to faces and audio not seen at training time. To achieve this we propose an encoder-decoder CNN model that uses a joint embedding of the face and audio to generate synthesised talking face video frames. The model is trained on tens of hours of unlabelled videos. We also show results of re-dubbing videos using speech from a different person.
1 Introduction
The paper addresses audio-to-video generation by introducing Speech2Vid, which synthesizes a target identity speaking from still face images and speech. It learns direct audio–video correspondences with a joint face-and-speech embedding and supports unseen identities, audio, and video re-dubbing.
- Speech2Vid generates a talking-face video from a target face image and an audio speech segment, even when the speaker differs from the target identity.
- The model learns correspondences between raw audio and video directly rather than learning phoneme-to-viseme mappings.
- A joint embedding of the target face and speech segment generates frames of that identity lip-synced to the speech.
- The paper applies the model to re-dubbing videos by visually blending generated faces into source video frames.
- Speech2Vid is trained with unsupervised learning on tens of hours of people talking and does not require labelled videos.
Dataset
The dataset pipeline prepares large-scale aligned face and speech samples from videos. It detects and tracks faces, identifies the active speaker, corrects audiovisual synchronization, registers faces canonically, and samples training frames.
- The preparation pipeline obtains tens of hours of visual face sequences aligned with spoken audio.
- The main stages detect and track face appearances, determine who is speaking, and align each detected face to a canonical face.
- VoxCeleb and LRW provide front-facing, high-quality interview and broadcast-news videos with clearly spoken words and limited background noise.
- Face tracking uses shot boundaries, a HOG-based DLIB detector, KLT track grouping, and regression-tree facial landmarks.
- SyncNet identifies the active speaker and corrects broadcast lip-sync errors to align mouth images with audio samples.
- Spatial registration applies scale, rotation, and translation using eye and nose landmarks while excluding the mouth.
3 The Speech2Vid Model
Speech2Vid combines audio and identity encoders with an image decoder and deblurring module to generate talking-face frames from speech and still identity images. It is trained with registered video frames and generates outputs frame-by-frame from sliding audio windows.
- Inputs and output: The model takes a 0.35-second audio segment and still target-identity images, then generates one corresponding talking-face frame.Video is produced by sliding the window across the audio sequence.
- Architecture: Speech2Vid contains audio and identity encoders, an image decoder, and a separately trained deblurring module.The decoder combines encoded audio and identity features to generate the image, while deblurring sharpens its output.
- Input representations: The identity encoder uses a 112 × 112 × 3 still image, while the audio encoder represents each 0.35-second sample as a 12 × 35 MFCC heatmap.The MFCC representation contains 35 time steps and 12 retained coefficients per step.
- Architecture: Two skip connections concatenate encoder and decoder activations to preserve defining facial features from the identity image.An L1 loss is used because it tends to encourage less blurring than L2 loss.
- Training: Training uses ground-truth middle frames as supervision while sampling identity images from different times in the same video stream.The training strategy supports learning from unlabelled videos and, with multiple inputs, samples several identity images from the stream.
- Runtime: At test time, the network including deblurring runs faster than twice real-time on a GPU.Identity features can be precomputed, and redubbed videos are generated at the source video's frame rate.
4 Experiments
Experiments visualize synchronized generated faces, test identity preservation with skip connections and multiple still images, and apply the model to video redubbing. The authors report improved identity quality with multiple images and favor blending only the lower face for naturalness.
- Qualitative results: Generated mouth movements in Figure 10 reflect the sounds of the highlighted words in the audio captions.The visualization shows generated frames for two identities and corresponding audio segments.
- 4.1 Preserving Identity with Skip Connections: Skip connections preserve defining identity features, whereas removing them causes generated faces to lose those features.Earlier skip connections were avoided because they made outputs too similar to the still image and restricted desired mouth shapes.
- 4.2 Preserving Identity with Multiple Still Images: Multiple still images enhance generated-face quality by providing more mouth configurations and reducing the influence of identity-irrelevant facial variations.The authors specifically compare one and five input images.
- 4.3 Application: Lip Transplant/Re-dubbing Videos: Video redubbing extracts identity stills, generates the new speech-conditioned face, realigns landmarks, and blends the result into source frames.Similarity transformation aligns faces, while Poisson editing matches gradients at blending boundaries.
- 4.3 Application: Lip Transplant/Re-dubbing Videos: Blending only the lower half of the face, from just below the eyes, gives the best reported balance between image naturalness and movement naturalness.The method can also blend the entire generated face.
5 Summary and extensions
The paper demonstrates Speech2Vid’s ability to generate talking-face videos from arbitrary identities and audio, including re-dubbing, while identifying quantitative evaluation as an open extension.
- Speech2Vid generates videos of any identity speaking from any source of input audio.
- The model also supports re-dubbing videos using an audio source independent of the original speaker.
- A clear extension is adding a quantitative performance measure for the models.
- Quantitative evaluation is difficult because generative models lack a definitive domain-specific performance measure.
- The model could be applied to computer facial animation relying only on audio.