Source-linked AI summary
Neural Voice Puppetry: Audio-driven Facial Reenactment
Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, Matthias Nießner
TL;DR
Neural Voice Puppetry addresses the lack of a visual channel for speech-based digital assistants by generating synchronized, photo-realistic facial video from audio. It maps audio into shared and person-specific 3D expression spaces, then renders the target appearance; the approach supports different source voices and synthetic speech, while requiring synchronized, occlusion-free target footage and failing with multiple voices.
Problem
Speech-based interaction is widespread, but its visual counterpart for appealing digital assistants is largely missing.
Method
The method maps audio through a shared latent audio-expression space into person-specific 3D blendshape expressions and uses lightweight neural rendering for the target face.
Results
The approach generalizes across different audio sources and supports photo-realistic video synthesis from both another person’s audio and synthesized voices.
Takeaways & Limitations
The system supports audio-driven avatars, facial reenactment, and text-driven talking-head video synchronized with artificial voice.
Takeaways & Limitations
The method fails for audio streams with multiple voices and requires occlusion-free, well-synchronized target videos; it also assumes a constant target talking style.
Abstract
from arXiv · showhide
We present Neural Voice Puppetry, a novel approach for audio-driven facial video synthesis. Given an audio sequence of a source person or digital assistant, we generate a photo-realistic output video of a target person that is in sync with the audio of the source input. This audio-driven facial reenactment is driven by a deep neural network that employs a latent 3D face model space. Through the underlying 3D representation, the model inherently learns temporal stability while we leverage neural rendering to generate photo-realistic output frames. Our approach generalizes across different people, allowing us to synthesize videos of a target actor with the voice of any unknown source actor or even synthetic voices that can be generated utilizing standard text-to-speech approaches. Neural Voice Puppetry has a variety of use-cases, including audio-driven video avatars, video dubbing, and text-driven video synthesis of a talking head. We demonstrate the capabilities of our method in a series of audio- and text-based puppetry examples, including comparisons to state-of-the-art techniques and a user study.
1 Introduction
Neural Voice Puppetry supplies a missing visual channel for speech-based agents by generating photo-realistic, audio-synchronized facial video from short target videos. Its architecture combines person-specific expression modeling with neural rendering for practical audio-to-video translation.
- Neural Voice Puppetry generates photo-realistic facial animation for visual digital assistants from audio input.
- The method estimates lip motions matching input audio and renders the target person’s appearance using aligned audio-image training data.
- A 2–3 min target video provides the person-specific talking style and appearance representation, avoiding the extensive data required by prior work.
- Audio2ExpressionNet maps audio to a 3D blendshape basis representing person-specific talking styles while generalizing across news speakers.
- A lightweight neural rendering network reproduces person-specific appearance in photo-realistic video and surpasses prior neural rendering methods in quality and speed.
2 Related Work
Related work separates facial animation, which predicts motions for predefined avatars, from facial reenactment, which seeks photo-realistic video of existing people. Neural Voice Puppetry targets the latter using audio input and short target videos.
- Facial animation predicts facial motions for predefined avatars, whereas facial reenactment generates photo-realistic video of an existing person with their idiosyncrasies.
- Prior audio-driven animation methods often produce motions or mesh coordinates rather than photo-realistic output and may require high-quality 3D reconstructions.
- Audio-driven facial reenactment seeks photo-realistic videos synchronized with input audio, but few methods generate natural full-frame imagery.
- Obama reenactment required 17h of speech data, whereas Neural Voice Puppetry requires only a 2–3 min target sequence.
- The pipeline uses audio features, a generalized network, person-specific expression mappings, and neural rendering to produce the target video.
3 Overview
Neural Voice Puppetry uses a shared audio-expression space to transfer speech-driven motion across people, then maps it into a target-specific expression space for neural rendering.
- A generalized network predicts latent expressions in an audio-expression space shared across people, enabling motion transfer between identities.
- Pretrained speech-to-text features provide audio representations for estimating expressions with input-audio generalization.
- Target-specific mappings interpret shared audio expressions as blendshape coefficients for the selected person’s 3D face model.
- The training corpus supplies samples used to optimize Audio2ExpressionNet.
4 Data
The method is built from synchronized in-the-wild videos, with preprocessing that extracts facial tracking and audio features for training a low-dimensional 3D facial representation.
- Training uses in-the-wild videos downloaded from the internet, provided their audio and visual content are synchronized.
- Experimental videos have 512 × 512 resolution at 25fps.
- Automatic preprocessing extracts face-tracking information and audio features required for training.
- A statistical face model with delta-blendshapes represents facial animation in a fixed latent space of 100 shape, 100 albedo, and 76 expression parameters.
5 Method
Neural Voice Puppetry maps generalized audio features into person-specific 3D facial expressions, then uses neural rendering to synthesize photo-realistic, temporally stable facial video. Training separates cross-person generalization from target-sequence specialization, while inference uses only source audio and runs in real time.
- Audio-to-expression representation: A shared latent audio-expression space maps audio to person-specific expression spaces through a 3D face model, enabling facial-motion reenactment across people.The model constrains person-specific audio-expression blendshapes as linear combinations of generic blendshapes.
- Audio-to-expression representation: Audio2ExpressionNet uses DeepSpeech features for per-frame expression estimation and content-aware temporal filtering to produce smooth audio-expressions.The filtering network combines neighboring per-frame estimates using expression-dependent weights.
- Training objectives: The expression network is trained with vertex-based RMS loss against tracked face geometry, emphasizing the mouth region by 10x and adding temporal displacement losses.Forward, backward, and central differences are weighted by λ = 20, with losses measured in millimeters.
- Neural rendering: A lightweight neural-texture renderer synthesizes the target’s lower face using separate interior and image-embedding networks, optimized with ℓ1 and VGG losses.The intermediate face-interior result and final synthetic image are both compared with the ground-truth image.
- Training and inference: Training has generalized and specialized phases: the first learns cross-person audio-expression mappings, while the second trains rendering for a specific target sequence.At test time, person-specific mappings can be obtained for unseen actors from visually estimated training expressions.
- Training and inference: Inference requires only source audio; audio-to-expression mapping takes less than 2ms, 3D reconstruction and rasterization another 2ms, and rendering approximately 5ms.These timings yield a real-time-capable pipeline, and text-driven synthesis can feed generated speech into the same audio-to-video process.
6 Results
The evaluation covers self-reenactment, cross-language transfer, comparisons with state-of-the-art methods, and a 56-person user study. Results assess reconstruction error, temporal prediction, synchronization, visual quality, and applicability with short training videos.
- Self-reenactment: The pipeline is evaluated through self-reenactment using ground-truth video and visual face tracking.The evaluation measures both the rendering network and the complete audio-based reenactment pipeline.
- Self-reenactment: The temporal prediction network reduces relative 2D mouth landmark error from 0.058 per-frame prediction to 0.055 temporal prediction.Temporal prediction also produces smoother predictions.
- Cross-language transfer: SyncNet measurements evaluate audio-visual synchronization across videos with different source and target languages, using original target videos as references.Higher confidence indicates better synchronization, while confidence below 1 indicates uncorrelated audio-video streams.
- Comparisons to state-of-the-art methods: A user study with 56 attendees rates the visual and audio-visual quality of videos from state-of-the-art methods.Participants judged 24 videos on five-point quality scales.
- Comparisons to state-of-the-art methods: The method is compared with image-based and model-based audio-driven approaches using visual quality and synchronization measurements.The comparisons use the same input audio streams and include SyncNet offset/confidence measurements.
- Generalization: The approach supports multiple target identities when only 2-3 min of target training data are available.The evaluation highlights transferability across different target persons and person-specific talking styles and appearances.
7 Limitations
The method has limitations involving audio mixtures, target-video occlusions, source synchronization, and the assumption of a constant target talking style.
- Audio conditions: The method fails when the audio stream contains multiple voices.The paper identifies this as the cocktail-party issue and notes that visual cues are used by recent work to address it.
- Input-video conditions: Good visual tracking requires target videos to be free of occlusions.The paper states this boundary applies to reenactment approaches generally.
- Input-video conditions: The original audio-visual synchronization must be good because synchronization errors transfer to reenactment quality.This constraint applies to the source target videos used for reenactment.
- Talking-style assumption: The method assumes that the target actor maintains a constant talking style throughout a target sequence.Follow-up work is proposed to estimate talking style from audio and adapt facial-motion expressiveness.
8 Conclusion
Neural Voice Puppetry generalizes audio-driven facial reenactment across audio sources, including synthesized voices, enabling synchronized talking-head video from text-to-speech audio.
- Conclusion: The approach generalizes across different audio sources for audio-driven facial reenactment.It can synthesize a target talking head from another person’s audio or from a synthesized voice.
- Conclusion: Text-driven video synthesis is achieved by generating synchronized facial video from artificial voice audio.The paper positions this capability toward photo-realistic audio-visual assistants.
A Network Architectures
The Audio2ExpressionNet predicts facial expressions from speech features in two stages, then temporally filters the predictions before producing the final expression representation.
- Audio2ExpressionNet: Audio2ExpressionNet first estimates per-frame expressions from DeepSpeech features as a 32-dimensional audio-expression vector.These predictions represent facial expressions based on audio.
- Audio2ExpressionNet: An expression-aware filtering network temporally smooths the per-frame audio-expression predictions.The filtering network can be trained jointly with the per-frame expression estimator.
- Filtering network: The filtering network receives T = 8 predicted audio expressions and maps them through successive 1D convolutions to filter weights.The feature dimensions reduce from 8 × 32 through 8 × 16, 8 × 8, 8 × 4, 8 × 2, and 8 × 1 before a fully connected layer.
- Training setup: The training mask is used for the Audio2ExpressionNet.The figure identifies the mask as part of the network’s training setup.
Rendering network:
The rendering pipeline converts expression-driven neural textures into photo-realistic target frames using a deferred neural renderer and an inpainting network.
- Rendering network:: Expression predictions drive a person-specific 3D face model whose neural texture is rendered into the target image space.The renderer converts sampled neural descriptors to RGB values before embedding the result into the target video frame.
- Rendering network:: The inpainting network blends the modified face interior into the target image.
- Rendering network:: The two rendering networks share a structure but accept different input dimensions: 16-channel neural descriptors versus a 6-channel compositing input.
- Rendering network:: The modified U-Net uses dilated convolutions instead of strided convolutions and standard convolutions instead of transposed convolutions.
- Rendering network:: Training uses Adam with a learning rate of 0.0001; Audio2ExpressionNet and rendering networks train for 50 epochs.
B User Study
The user study compares videos from state-of-the-art methods and original footage by rating visual quality and audio-visual alignment, including lip sync.
- B User Study: The study contains 24 videos from different state-of-the-art methods, including three original videos.
- B User Study: The reported statistics summarize average answers and variance across the study videos.
- B User Study: Participants rate each video on audio-visual alignment and visual quality.
- B User Study: Responses use five categories ranging from “very good” to “very bad.”
- B User Study: The comparison includes Chen et al. and Vougioukas et al. using publicly available pretrained models on an Obama self-reenactment sequence.The sequence provides ground-truth images for comparison.
Image-based & Audio-driven Facial Animation:
The paper contrasts audio-driven facial reenactment with image- and video-driven alternatives, emphasizing photo-realistic full-frame output, target-style preservation, and practical as well as ethical implications.
- Image-based & Audio-driven Facial Animation:: Compared with Chen et al. and Vougioukas et al., the method is reported to produce higher visual image quality and full-frame images.
- Image-based & Audio-driven Facial Animation:: Unlike Voca, the method targets photo-realistic videos synchronized with audio and uses a 3D proxy rather than requiring a 4D training corpus.
- Image-based & Audio-driven Facial Animation:: Unlike video-driven dubbing systems, the method relies only on the dubber’s voice and operates directly in the target actor’s expression space.
- Image-based & Audio-driven Facial Animation:: The method is described as handling strong expressions, whereas Fig. 16 illustrates that visual dubbing can fail to map them plausibly.
- Image-based & Audio-driven Facial Animation:: Fried et al.’s text-based editing uses transcript and audio and synthesizes selected new-word frames, while this method uses synthetic audio and generates the entire video.
- Image-based & Audio-driven Facial Animation:: Combining the method with person-specific audio generators can create video-realistic content, but such systems can be misused and are difficult for humans to detect in compressed video.