Source-linked AI summary
Everybody's Talkin': Let Me Talk as You Want
Linsen Song, Wayne Wu, Chen Qian, Ran He, Chen Change Loy
TL;DR
Audio-based portrait editing must generate realistic target videos from arbitrary source audio despite limited target footage, identity differences, and the difficulty of direct audio-to-video mapping. The paper factorizes target frames, translates audio into expression parameters, preserves target geometry and pose, and renders the result with landmark-conditioned video completion. Extensive experiments demonstrate robustness and effectiveness, while strong accents and mismatched driving emotions remain limitations.
Problem
The paper addresses audio-to-video portrait editing across arbitrary source audios and target videos with scarce target footage, without assuming a shared identity.
Method
The method translates source audio into expression parameters, combines them with target geometry and pose, and uses identity removal plus landmark-conditioned rendering to synthesize coherent portrait video.
Results
Extensive experiments demonstrate the method’s robustness and the effectiveness of its pivotal components for generating photo-realistic videos.
Takeaways & Limitations
An end-to-end learnable framework can edit portrait video using arbitrary audio sources and targets without a person-specific rendering network.
Takeaways & Limitations
The method performs poorly for strongly accented driving speech and produces unnatural results when driving-audio emotion differs from the source video’s emotion.
Abstract
from arXiv · showhide
We present a method to edit a target portrait footage by taking a sequence of audio as input to synthesize a photo-realistic video. This method is unique because it is highly dynamic. It does not assume a person-specific rendering network yet capable of translating arbitrary source audio into arbitrary video output. Instead of learning a highly heterogeneous and nonlinear mapping from audio to the video directly, we first factorize each target video frame into orthogonal parameter spaces, i.e., expression, geometry, and pose, via monocular 3D face reconstruction. Next, a recurrent network is introduced to translate source audio into expression parameters that are primarily related to the audio content. The audio-translated expression parameters are then used to synthesize a photo-realistic human subject in each video frame, with the movement of the mouth regions precisely mapped to the source audio. The geometry and pose parameters of the target human portrait are retained, therefore preserving the context of the original video footage. Finally, we introduce a novel video rendering network and a dynamic programming method to construct a temporally coherent and photo-realistic video. Extensive experiments demonstrate the superiority of our method over existing approaches. Our method is end-to-end learnable and robust to voice variations in the source audio.
1. Introduction
The paper targets audio-based editing of portrait footage without reshooting or relying on highly skilled editors. It addresses the difficulty of arbitrary audio-to-video translation by mapping audio to expression parameters while retaining target geometry and pose, then rendering coherent, realistic frames.
- Audio-based portrait editing is motivated by applications in filmmaking, video production, and telepresence, while conventional editing can require reshooting footage and depends heavily on editor expertise.
- Prior audio-driven methods either animate a still image, target stylized avatars, or assume matching source and target identities with extensive single-identity training data.The cited Obama-focused method requires up to 17 hours of single-identity data.
- The proposed framework performs many-to-many audio-to-video translation with scarce target footage, such as at most 15 minutes per person.It is designed for arbitrary source audios and target videos despite limited data and large audio-video variation across subjects.
- Audio is translated into expression parameters instead of full pixels, because expression is more semantically related to audio than geometry or pose.The generated expression is combined with the target portrait’s geometry and pose to preserve identity and head pose while changing phoneme-matched lip movements.
- An Audio ID-Removing Network supports voice variation, while a landmark-conditioned rendering network completes masked mouth regions to generate photo-realistic video.The renderer uses landmark heatmaps and masked target frames to guide frame-wise mouth completion.
- The contributions include an end-to-end learnable audio-based portrait-editing framework, identity-agnostic audio-to-expression translation, and a single network for rendering arbitrary people.
2. Related Work
Related work spans speech-driven facial animation, photorealistic portrait synthesis, model-based reenactment, and generative methods. Existing approaches face limitations involving still-image animation, identity specificity, data requirements, person-specific models, or alignment artifacts.
- Audio-based Facial Animation: Speech-driven facial animation associates phonemes or speech features with visemes across face rigs, meshes, and expression blendshapes.
- Audio-based Facial Animation: Photorealistic portrait video is harder than 3D head animation because speaker-specific appearance and head pose strongly affect generated-video quality.Using a still face as reference can improve visual quality but produces animation of a still image rather than a natural video.
- Audio-based Facial Animation: The Obama portrait method achieves state-of-the-art results but assumes source and target identities match and requires up to 17 hours of training data.These requirements limit its applicability to different voices and target actors with scarce footage.
- Video-based Facial Reenactment: Model-based methods disentangle geometry, expression, and pose, but person-specific designs and audio-visual misalignment can create artifacts and unrealistic results.
3. Methodology
The method factorizes target portrait frames into geometry, expression, and pose, translates identity-removed audio into expression parameters, and renders temporally coherent mouth motion while retaining target-video context.
- 3D Face Modeling: The pipeline extracts geometry, pose, and expression parameters from each target-video frame using a parametric 3D face model.These parameters are recovered through constrained nonlinear optimization; geometry and expression bases are reduced with PCA.
- Audio ID-Removing Network: An Audio ID-Removing Network transfers speaker-dependent MFCC features toward a global-speaker domain before audio-to-expression translation.An LSTM+FC network infers adaptation coefficients, while a VGGVox-based cross-entropy constraint makes the transformed MFCC spectrum less speaker-identifiable.
- Audio-to-Expression Translation: The Audio-to-Expression Translation Network maps a one-second identity-removed speech window to facial expression parameters and jointly uses expression and landmark losses.The window contains 0.8 seconds before and 0.2 seconds after the target time; the shape constraint supervises reconstructed facial landmarks.
- Neural Video Rendering: A completion-based Neural Video Rendering Network generates the mouth region from landmark heatmaps while conditioning on dynamic background portrait frames.A tailored retiming-based dynamic program selects frames, and a modified U-Net transfers fine-scale structure through skip connections.
- Neural Video Rendering: The renderer combines seven RGB frames and seven heatmap images, then uses blending and temporal flicker removal to improve continuity.Mouth masks are formed from landmark polygons, softened, and used with Poisson blending; a temporal algorithm addresses lip-motion jitter and appearance flicker.
- Loss Functions: The rendering objective combines reconstruction, adversarial, perceptual, total-variation, and gradient-penalty terms within end-to-end training.The total objective is Ltotal = Lnorm + Ltrans + Lrender, with Lrender = Lrecon + Ladv + Lvgg + Ltv + Lgp.
4. Results
The method is evaluated across many-to-many speaker mappings, large head poses, audio editing, singing, state-of-the-art comparisons, ablations, and a user study. Results support generalized audio-to-video translation, improved parameter regression and identity removal, effective joint training, and realistic generated videos.
- Audio-to-Video Translation: One-to-many and many-to-one experiments demonstrate audio-to-expression translation across different speakers using a shared generator rather than person-specific generators.The experiments drive three speakers with one speaker's audio and one speaker with audio from two different speakers.
- Audio-to-Video Translation: The method generates portrait videos under varied large head poses because audio is associated with expression parameters rather than shape or pose parameters.Figure 5 demonstrates up, down, right, and left poses, while results for all seven poses are available on the project page.
- Audio-to-Video Translation: Audio editing recombines selected source-audio phrases, and the network is also evaluated on singing clips.The reported editing example forms “Knowledge is virtue” from two phrases, while Figure 7 evaluates singing audio.
- Comparison with State-of-the-Art: Comparisons with Audio2Obama, Face2Face, DVP, and TBE report sharper texture details, accurate lip movement, and seamless blending with dynamic backgrounds.The comparisons attribute texture improvements to the GAN-based rendering network and contrast the method's original-background blending with DVP's potential failure in complex dynamic backgrounds.
- Ablation Study: 3D parameter regression is evaluated against landmark regression, while an Audio ID-Removing Network and completion-based generation are evaluated through quantitative and qualitative comparisons.The studies include expression, landmark, PSNR, and SSIM measures; identity-removal results assess lip synchronization; jointly trained generators outperform separately trained person-specific generators with fewer parameters as speaker count increases.
5. Conclusion
The method performs end-to-end audio-based video editing by translating audio into expression parameters and rendering photo-realistic videos. It combines identity-agnostic audio processing, landmark-guided generation, and temporal processing to improve robustness and continuity.
- Conclusion: The framework maps audio to expression space instead of directly mapping audio to video, then introduces identity removal and neural video rendering for arbitrary targets and audio sources.This bypasses the highly nonlinear direct mapping and supports photo-realistic video generation.
- Conclusion: During training, paired audio and video frames supervise predicted expression parameters derived from monocularly reconstructed facial shape, expression, and pose.The Audio-to-Expression Translation Network uses ground-truth expression parameters as supervision.
- Conclusion: The temporal flicker-removal process smooths landmark coordinates and reduces appearance flicker, although adjacent audio clips already provide acceptable frame continuity.The method generates frames independently, while overlapping audio clips contribute to continuity across neighboring frames.
- Conclusion: The appearance deflicker algorithm weights the previous frame according to mouth motion rather than global frame consistency.The previous-frame weight is λ_t = exp(−d_t), with optical-flow warping used to align the preceding processed frame.
C.1. Quantitative Comparison on GRID dataset
The GRID-dataset comparison evaluates the proposed method against recent full-face generation methods using image-generation metrics. The architecture includes audio-to-expression translation, identity removal, and landmark-guided face completion.
- C.1. Quantitative Comparison on GRID dataset: The proposed architecture registers a parametric 3D face model to extract geometry, pose, and expression, then maps source audio features to expression parameters.An Audio ID-Removing Network addresses speaker variation, while mouth landmarks guide face completion.
- C.1. Quantitative Comparison on GRID dataset: The comparison uses SSIM, PSNR, IS, and FID to evaluate the proposed method against recent methods on the GRID dataset.For fairness, the comparison generates the full face without post-processing.
C.2. Ablation Study on Temporal Flicker Removal Algorithm
The ablation study finds that temporal flicker removal produces only slight quantitative gains but visibly improves temporal continuity, especially when the mouth remains closed.
- C.2. Ablation Study on Temporal Flicker Removal Algorithm: The temporal flicker removal algorithm yields slight quantitative improvement but visibly improves continuity, particularly when the mouth does not open.The study evaluates the 3D parameter regression baseline, Audio ID-Removing Network, and temporal flicker removal algorithm in Table 6.
C.3. Audio ID-removing Effects in tSNE map
The tSNE visualization indicates that speaker identity becomes indistinguishable in the MFCC spectrum after processing by the Audio ID-Removing Network.
- C.3. Audio ID-removing Effects in tSNE map: After identity removal, the tSNE map shows that speaker identity cannot be distinguished in the MFCC spectrum.The visualization compares input MFCC features with identity-removed MFCC features.
D. Runtime Performance
The system generates video frames in real time without post-processing, while optical-flow-based deflickering substantially increases inference time. The reported component timings identify the rendering and post-processing costs.
- Component evaluation: Table 6 reports that ID-removing and deflickering components are evaluated quantitatively, with metrics validating the proposed components except deflickering.The table notes that deflickering primarily removes temporal discontinuity visible in the video.
- Runtime breakdown: 77 ms per frame is required by the inpainting network, compared with 17 ms per frame for audio-to-expression translation.
- Post-processing: 1.3s per frame is spent on deflickering, whose optical-flow calculation dominates inference time.
- Post-processing: 300 ms per frame is required by the teeth-proxy post-processing step.
E. Limitations
The method has limitations in modeling emotion, tongue movement, and accented speech. These limitations can produce unnatural expressions, poor tongue generation, or unsatisfactory lip synchronization.
- Emotion: The method does not explicitly model facial emotion or estimate sentiment, so mismatched driving-audio emotion can make the generated video look unnatural.
- Tongue: The rendering pipeline ignores tongue movement, causing poor tongue-texture generation for phonemes such as “Z”.Lip fiducials and a teeth proxy provide the modeled mouth details, but do not capture tongue texture well.
- Accent: Strongly accented driving speech can produce visually unsatisfactory lip-sync accuracy, as shown for a strong Russian accent.The authors attribute this to accented English speech being an outlier for the Audio-to-Expression Translation Network.