Source-linked AI summary
VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wild
Kun Cheng, Xiaodong Cun, Yong Zhang, Menghan Xia, Fei Yin, Mingrui Zhu, Xuan Wang, Jue Wang, Nannan Wang
TL;DR
VideoReTalking targets generic audio-based editing of talking-head videos, where existing methods may blur faces, misalign lips, require speaker-specific training, or lack emotion editing. It separates expression normalization, audio-driven lip synchronization, and identity-aware enhancement into a sequential pipeline, and reports higher visual quality with accurate lip synchronization on benchmark and in-the-wild videos. The authors also identify identity changes, artifacts in extreme poses, and temporal jittering as remaining limitations.
Problem
Existing generic visual-dubbing methods can produce blurry lower faces or inaccurate lip synchronization, while speaker-specific methods require target-speaker training and prior methods lack emotion editing.
Method
The system sequentially normalizes facial expression, synthesizes audio-driven lip motion from canonical references, and enhances the resulting face with an identity-aware network.
Results
The system produces higher visual quality with accurate lip synchronization on benchmark and in-the-wild videos.
Takeaways & Limitations
The generic pipeline works without fine-tuning and supports high-quality audio-based editing, with potential for emotional talking-head generation.
Takeaways & Limitations
The method can cause slight identity changes, artifacts in extreme poses, and small temporal jittering or flashing because processing is frame by frame.
Abstract
from arXiv · showhide
We present VideoReTalking, a new system to edit the faces of a real-world talking head video according to input audio, producing a high-quality and lip-syncing output video even with a different emotion. Our system disentangles this objective into three sequential tasks: (1) face video generation with a canonical expression; (2) audio-driven lip-sync; and (3) face enhancement for improving photo-realism. Given a talking-head video, we first modify the expression of each frame according to the same expression template using the expression editing network, resulting in a video with the canonical expression. This video, together with the given audio, is then fed into the lip-sync network to generate a lip-syncing video. Finally, we improve the photo-realism of the synthesized faces through an identity-aware face enhancement network and post-processing. We use learning-based approaches for all three steps and all our modules can be tackled in a sequential pipeline without any user intervention. Furthermore, our system is a generic approach that does not need to be retrained to a specific person. Evaluations on two widely-used datasets and in-the-wild examples demonstrate the superiority of our framework over other state-of-the-art methods in terms of lip-sync accuracy and visual quality.
1 INTRODUCTION
VideoReTalking addresses generic talking-head video editing by separating expression normalization, audio-driven lip synchronization, and face enhancement. The sequential system targets accurate lip sync and higher visual quality without speaker-specific retraining or manual intervention.
- Visual dubbing supports applications such as language translation and post-recording speech modification, but existing generic methods can produce blurry faces, inaccurate synchronization, and no emotion editing.
- The system first neutralizes facial expressions, then uses the modified frames as canonical pose references for more stable lip generation.Original frames can make lip generation sensitive to expression references and produce out-of-sync results.
- Expression reenactment generates a video with the same canonical expression across frames before lip synchronization.The method extracts 3DMM pose and expression coefficients, replaces expression with a neutral template, and regenerates the video.
- The lip-sync network synthesizes the lower-half face from audio and canonical references, while the enhancement network improves photo-realism.The lip-sync model uses an hourglass-like architecture with Fast Fourier Convolution, and audio features are injected through AdaIN.
- All modules run sequentially without manual intervention or fine-tuning, and experiments report higher visual quality with accurate lip synchronization.
2 RELATED WORK
Related work spans generic visual dubbing, personalized speaker-specific systems, and audio-conditioned single-image face animation. These approaches differ in whether they model video editing, require speaker retraining, or generate animation from a single image.
- Visual dubbing edits an input video according to speech audio while preserving other motions, whereas single-image animation generates facial motion from audio conditions.
- Generic methods aim to work across identities without retraining, while personalized methods are limited to a specific person and known environment.
- Prior generic systems include inpainting-based approaches that reconstruct the lower-half face, while other methods predict landmarks, 3DMM coefficients, or dense motion fields.
- Personalized visual dubbing methods commonly map audio to landmarks or expression coefficients before rendering photo-realistic video.
3 FRAMEWORK
The framework combines expression reenactment, audio-conditioned lip synthesis, and identity-aware enhancement to edit the lower-half face. Its design addresses reference-frame information leakage, low-resolution training data, and restoration artifacts.
- 3 FRAMEWORK: VideoReTalking is a cross-modal video-inpainting framework that fills the masked lower-half face using audio and emotion-modulated reference frames.
- 3.1 Semantic-guided Reenactment Network: D-Net replaces original expressions with a fixed template so stabilized frames can serve as structure references for L-Net.The pipeline extracts pose and expression coefficients, substitutes the expression coefficient, and synthesizes frozen-expression video.
- 3.1 Semantic-guided Reenactment Network: Using original frames directly leaks source lip motion into synthesis, causing generated lips to follow the reference even when audio should control them.
- 3.1 Semantic-guided Reenactment Network: Expression templates such as smiles can produce emotional talking faces, with upper-face editing supporting emotions including anger and surprise.
- 3.2 Lip-Sync Network: L-Net uses D-Net frames as identity and structure references, with audio and masked original frames as conditions for frame-wise lip synthesis.Its overview contains separate audio and video processing subnetworks.
- 3.3 Identity-aware Enhancement Network: Identity-aware enhancement is introduced because low-resolution training data limits visual quality and direct GAN-prior restoration can change identity or blur teeth and faces.The enhancement network is trained using restored high-resolution data and addresses the domain gap between training inputs and blurry L-Net outputs.
4 TRAINING
The system trains its modules individually and runs them sequentially without manual intervention, using datasets and metrics spanning low- and high-resolution talking-head video.
- Each module is trained individually, then the complete framework is tested sequentially without manual intervention.
- D-Net: D-Net is trained on VoxCeleb, which contains 22,496 talking-head videos with diverse identities and head poses.
- L-Net: L-Net is trained on 160p LRS2 videos with perceptual loss and a lip-sync discriminator for visual quality and audio-visual synchronization.
- E-Net: E-Net is trained for 300k iterations on an enhanced high-resolution LRS2 dataset.
- Evaluation: Evaluation measures visual quality with FID and CPBD and lip synchronization with LSE-C and LSE-D on LRS2 and HDTF.HDTF contains 720p or 1080p YouTube videos, while visual metrics are computed on full video frames.
5 RESULTS
Experiments compare VideoReTalking with LipGAN, Wav2Lip, and PC-AVS, showing strong visual and lip-sync quality, while also examining component contributions, emotional editing, and failure cases.
- 5.1 Comparison with state-of-the-art Methods: VideoReTalking achieves much better visual quality according to CPBD and FID, while maintaining much better and comparable lip-sync performance across the two datasets.The authors attribute Wav2Lip’s better LRS2 FID to that dataset’s low resolution and VideoReTalking’s high-resolution output.
- 5.1 Comparison with state-of-the-art Methods: The user study evaluates visual and lip-sync quality for ten HDTF videos generated by VideoReTalking, LipGAN, and Wav2Lip.Users rate each video on five scores, with larger values indicating better quality.
- 5.2 Ablation Study: Cross-attention improves lip-sync quality and vividness, E-Net improves visual quality but adds artifacts, and D-Net improves both video and lip-sync quality.
- 5.3 Extensions to Emotional Talking Video: The framework supports emotional talking-head editing for the lower-half face, combined with upper-face editing to create different emotional combinations.
- 5.4 Limitation: Failure cases include slight identity changes from D-Net’s dense warping and artifacts in extreme poses.
- 5.4 Limitation: Frame-by-frame editing can produce small temporal jittering and flashing.
6 CONCLUSION
The paper concludes that VideoReTalking is a generic, audio-conditioned system for in-the-wild talking-head editing that operates without fine-tuning, while acknowledging misuse concerns and testing expression-template effects.
- The framework removes lip motion before editing and produces high-quality in-the-wild results without fine-tuning.
- The system has potential for emotional talking-head generation in the lower-half face.
- Ethical Considerations: Because the system can edit talking content in the wild, the authors identify misuse concerns and propose robust video and audio watermarks and trustworthiness tools.
- Interpolating expression templates between neutral and smile produces only minor changes in lip-sync metrics on ten HDTF videos.
B ANALYSIS OF THE TRADE-OFF BETWEEN IDENTITY PRESERVATION AND EXPRESSION ANIMATION
D-Net exposes a trade-off between identity preservation and expression animation, with one-shot reenactment favoring stable lip motion and video-to-video processing favoring identity preservation.
- Trade-off: One-shot reenactment produces more stable lip animation and better lip-sync performance but slightly modifies identity through dense-flow warping.The reference frame is fixed and warped using the original pose and template-expression coefficients.
- Trade-off: Video-to-video reenactment better preserves identity but can introduce slightly unstable lip movement, so it is selected as the default setting.The reference frame is not fixed, allowing D-Net to change each frame’s expression.
- Trade-off: The trade-off arises because the two settings exploit information leakage between models differently during expression editing.The paper identifies information leaks between models as useful for expression editing.
C.1 Implementation Details of D-Net
D-Net combines coefficient mapping, feature warping, and refinement to perform semantic-guided expression reenactment, trained with perceptual and style-related objectives on VoxCeleb.
- Model architecture: D-Net uses three sub-networks for coefficient mapping, feature warping, and refinement in an encoder-decoder architecture with skip connections.The mapping network translates driven 3DMM coefficients into latent codes that are injected into the warping and refinement networks.
- Model architecture: AdaIN blocks inject motion information after convolution layers, while the warping network generates dense flow fields at one-quarter of the original image size.The editing network learns multi-scale features through three stages.
- Loss functions: The warping network is optimized with VGG-19 perceptual loss comparing warped images with ground truth.The feature-map layer used in the loss is defined through the pre-trained VGG-19 network.
- Loss functions: The editing network uses perceptual loss and Gram-matrix style loss between generated images and ground truth.The Gram matrix is constructed from an activation map.
- Training: D-Net is trained on VoxCeleb with self-reconstruction pairs, using 200k iterations for mapping and warping pretraining followed by 200k iterations for whole-network training.Training uses randomly selected source and target frames from the same video.
C.2.1 Model Architecture.
L-Net combines separate audio and visual encoders with cross-attention and FFC-based decoding to synthesize synchronized lower-half faces while preserving visual information from references.
- Model architecture: Cross-attention fuses original-frame queries and keys with reference-frame values to model long-range dependencies and reduce local information leakage.The fused visual features feed the decoder.
- Model architecture: The visual decoder uses three up-sampling layers, nine Modulated Res-FFC blocks, and skip connections from the visual encoder.The architecture uses Fast Fourier Convolution blocks as described in the component overview.
- Training objective: L-Net is trained with RGB L1 and feature-space perceptual losses for visual quality, plus a SyncNet loss for audio-visual synchronization.The synchronization loss is computed over continuous five-frame windows.
- Training objective: The synchronization-loss weight is λ_sync = 0.3, while the RGB and perceptual-loss weights are λ1 = 1 and λp = 1.These weights define the full L-Net optimization objective.
- Inference: During inference, the whole input sequence serves as the reference to preserve pose and background information.Training uses five continuous frames and randomly selected references from the same video.
C.3.1 Model Architecture.
E-Net enhances L-Net outputs through identity encoding and style-based super-resolution, using reconstruction, perceptual, identity, and adversarial objectives.
- Model architecture: E-Net contains an identity encoder E_i-Net and a super-resolution module E_u-Net for upsampling generated videos.The architecture processes continuous frames together with randomly selected same-video references.
- Model architecture: E_u-Net uses StyleGAN-inspired blocks to upsample low-resolution lip-synchronized results after they are generated by L-Net.The pipeline downsamples augmented high-resolution inputs before sending them through the pre-trained L-Net.
- Identity preservation: The identity encoder extracts a 512-dimensional high-level feature from resized high-resolution reference frames.It uses six down-sampling layers followed by a linear layer.
- Loss functions: E-Net combines RGB L1, perceptual, identity, and adversarial losses to train high-resolution outputs.The identity loss uses a pre-trained ArcFace network, while the adversarial loss targets realistic generated samples.
- Training: The loss weights are λ1 = 0.2, λp = 1, λadv = 100, and λid = 0.4.Training data are produced by applying GPEN face restoration to LRS2 and generating 96 × 96 low-resolution inputs with differentiable JPEG and bilinear downsampling.