Source-linked AI summary
ObamaNet: Photo-realistic lip-sync from text
Rithesh Kumar, Jose Sotelo, Kundan Kumar, Alexandre de Brebisson, Yoshua Bengio
TL;DR
The paper targets the limited work combining image generation and speech synthesis, and presents a system that generates speech and synchronized mouth movement from text. It combines text-to-speech, audio-conditioned keypoint prediction, and image-to-image video generation; consistent keypoints permit independent frame synthesis. The stated scope is people for whom suitable data is available.
Problem
The paper identifies little work that models image generation and speech synthesis together to generate video from text.
Source: §1 · p. 1Method
The system combines Char2Wav speech synthesis, an LSTM that predicts mouth shape from audio features, and pix2pix video generation conditioned on mouth shape.
Source: §3.1 · p. 2, §3.2 · p. 2, §3.3 · p. 2Results
The generated keypoints are consistent across time, allowing video frames to be synthesized independently without an explicit temporal-consistency mechanism.
Source: §3.3 · p. 2Takeaways & Limitations
The approach generates speech from arbitrary text and modifies an existing video's mouth area to look natural and realistic.
Source: §1 · p. 1The approach can be used for anyone only provided the data is available.
Source: §1 · p. 1
Abstract
from arXiv · showhide
We present ObamaNet, the first architecture that generates both audio and synchronized photo-realistic lip-sync videos from any new text. Contrary to other published lip-sync approaches, ours is only composed of fully trainable neural modules and does not rely on any traditional computer graphics methods. More precisely, we use three main modules: a text-to-speech network based on Char2Wav, a time-delayed LSTM to generate mouth-keypoints synced to the audio, and a network based on Pix2Pix to generate the video frames conditioned on the keypoints.
1 Introduction
The paper addresses the limited work combining image generation and speech synthesis, presenting a system that generates speech from text and adapts an existing video’s mouth. It presents Obama as an example while stating the approach can be used for anyone when data is available.
1 Introduction
- The work combines speech synthesis and image generation to produce a video of a person reading arbitrary text.The output modifies the mouth area of an existing video to look natural and realistic.Source: §1 · p. 1
- The system is trained on close-shot videos of a person speaking and their corresponding transcripts.Source: §1 · p. 1
- Although demonstrated on Barack Obama, the approach can be used for anyone provided the data is available.Source: §1 · p. 1
2 Related Work
Prior work had advanced photo-realistic video generation and audio-based facial animation, but the paper identifies Suwajanakorn et al. as its closest comparison. It distinguishes its approach by using a neural network and adding text-to-speech for text-to-video generation.
2 Related Work
- Earlier work made progress in photo-realistic video generation, including facial animation based on audio.Source: §2 · p. 1
- Suwajanakorn et al. (2017) is identified as the closest work, with this paper distinguishing itself by using a neural network instead of a traditional computer vision model.Source: §2 · p. 1
- Unlike the closest work, this approach adds a text-to-speech synthesizer to form a full text-to-video system.Source: §2 · p. 1
- Figure 1 is a flow diagram of the generation system.Source: Figure 1 · p. 1
3 Model Description
The model combines text-to-speech, audio-conditioned mouth-shape prediction, and image-to-image video generation. Its recurrent keypoints are consistent across time, allowing frames to be synthesized independently.
3.1 Text-to-speech system
- Char2Wav generates speech from input text and is trained using video-extracted audio with corresponding transcripts.Source: §3.1 · p. 2
3.2 Keypoint generation
- A time-delayed LSTM predicts mouth-shape representation from audio features, using normalized mouth keypoints reduced to prominent PCA components.Normalization makes keypoint generation compatible with any target video.Source: §3.2 · p. 2
3.3 Video generation
- Pix2pix translates a mouth-cropped face image into an in-painted face conditioned on mouth shape, implicitly supplied as an outline on the input.Source: §3.3 · p. 2
- Consistent keypoints allow frames to be generated independently without an explicit mechanism for temporal consistency.Source: §3.3 · p. 2
- The video-generation network is trained with pixel-space L1 loss, which the authors found sufficient without pix2pix’s additional GAN objective.Source: §3.3 · p. 2
4 Supplementary Material
The supplementary material describes the Obama video dataset and processing used to prepare speech, keypoint, and video-generation data. PCA reduces normalized mouth-keypoint features to a compact representation.
4.1 Dataset
- The demonstration uses 17 hours of footage from 300 weekly presidential addresses, filmed in a relatively controlled setting with Obama centered in the frame.Source: §4.1 · p. 3
4.2 Data Processing
- The speech data uses video-extracted audio converted to 16KHz, WORLD vocoder frames, and associated transcripts.Source: §4.2 · p. 3
- The keypoint data uses 68 facial landmarks, mean-normalized around the mouth, rotated to remove in-plane rotation, and divided by a norm approximating face size.Source: §4.2 · p. 3
- The first 5 PCA coefficients capture >98% variability in the 20 normalized keypoints represented as a 40-D vector.Source: §4.2 · p. 3
- Video-generation training pairs use a mouth-cropped face annotated with a mouth outline as input and the complete face as output.The mouth outline is drawn from keypoints 49–68; keypoints are denormalized using the target video's mouth location, size, and rotation for visual compatibility.Source: §4.2 · p. 3