Source-linked AI summary
One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learning
Suzhen Wang, Lincheng Li, Yu Ding, Xin Yu
TL;DR
One-shot talking-face methods trained across speakers can produce unnatural mouth shapes and poor lip synchronization. This paper learns audio-visual motion correlations from a single speaker and transfers them to reference images; it reports more natural mouths and accurate synchronization for unseen speakers than prior one-shot approaches.
Problem
One-shot methods trained on multiple speakers struggle to learn a consistent speaking style, and their results can have unnatural lip shapes and poor synchronization.
Source: Introduction · p. 1Method
The framework learns audio-visual correlations from one speaker with AVCT, then transfers keypoint-based motion fields to reference images for talking-face generation.
Source: Introduction · pp. 1–2Results
The method reports more accurate lip synchronization and more natural mouth shapes than prior one-shot approaches, and performs on par with speaker-specific methods for the training speaker.
Source: Introduction · p. 2Takeaways & Limitations
The reported scope includes arbitrary speakers and non-photorealistic paintings, in addition to photo-realistic talking-face videos.
Source: Conclusion · p. 7
Abstract
from arXiv · showhide
Audio-driven one-shot talking face generation methods are usually trained on video resources of various persons. However, their created videos often suffer unnatural mouth shapes and asynchronous lips because those methods struggle to learn a consistent speech style from different speakers. We observe that it would be much easier to learn a consistent speech style from a specific speaker, which leads to authentic mouth movements. Hence, we propose a novel one-shot talking face generation framework by exploring consistent correlations between audio and visual motions from a specific speaker and then transferring audio-driven motion fields to a reference image. Specifically, we develop an Audio-Visual Correlation Transformer (AVCT) that aims to infer talking motions represented by keypoint based dense motion fields from an input audio. In particular, considering audio may come from different identities in deployment, we incorporate phonemes to represent audio signals. In this manner, our AVCT can inherently generalize to audio spoken by other identities. Moreover, as face keypoints are used to represent speakers, AVCT is agnostic against appearances of the training speaker, and thus allows us to manipulate face images of different identities readily. Considering different face shapes lead to different motions, a motion field transfer module is exploited to reduce the audio-driven dense motion field gap between the training identity and the one-shot reference. Once we obtained the dense motion field of the reference image, we employ an image renderer to generate its talking face videos from an audio clip. Thanks to our learned consistent speaking style, our method generates authentic mouth shapes and vivid movements. Extensive experiments demonstrate that our synthesized videos outperform the state-of-the-art in terms of visual quality and lip-sync.
Introduction
The paper addresses unnatural mouth shapes and poor lip synchronization in one-shot generation by learning a consistent audio-visual speaking style from one speaker. It transfers that style to reference images and reports improved lip synchronization and mouth shapes for unseen speakers.
Introduction
- The paper motivates photo-realistic audio-driven portraits with applications including digital-human animation, visual dubbing, and short-video creation.These examples are given as applications of audio-driven photo-realistic portrait synthesis.Source: Introduction · p. 1
- Multi-speaker training can leave one-shot methods with unnatural lip shapes and poor synchronization as they fit a common speech style across identities.The paper attributes this difficulty to treating personalized speech-style variation as noise.Source: Introduction · p. 1
- The framework learns consistent audio-visual motion correlations from one speaker, then transfers audio-driven keypoint-based motion fields to a reference image.The approach is presented as learning authentic lip movements and rhythmic head motions from an individual before transferring them to multiple people.Source: Introduction · pp. 1–2
- The method reports more accurate lip synchronization and more natural mouth shapes than prior one-shot approaches, while producing videos for unseen speakers.It also generates training-speaker videos on par with speaker-specific methods and handles translational and rotational head movements.Source: Introduction · p. 2
- AVCT uses phonemes and facial keypoint-based motion representations to support extension to other audio and identities.The method also uses acoustic features to modulate mouth shapes and relative motion transfer to reduce motion gaps between differently shaped faces.Source: Introduction · p. 2
- The framework generates photo-realistic talking-face videos with natural mouth shapes, accurate lip synchronization, and rhythmic head motions from a reference image and new audio.The paper reports this outcome despite learning its audio-visual correlations from a specific speaker.Source: Introduction · p. 2
Related Work
Prior talking-face work includes speaker-specific systems and universal speaker-arbitrary systems. The paper situates its approach against their retraining requirements and difficulties with natural poses, speaking style, and lip quality.
Related Work
- The literature distinguishes speaker-specific methods from speaker-arbitrary methods that aim to build one universal model for multiple subjects.Earlier work also includes 3D face-model animation and 2D photo-realistic talking-face generation.Source: Related Work · p. 2
Speaker-specific Talking Face Generation
- Speaker-specific systems retrain part or all of their models for a new speaker, and many synthesize videos guided by 3D face models.The surveyed work also includes mouth-region synthesis, speaker-independent components with speaker-specific rendering, text-based editing, and neural radiance fields.Source: Speaker-specific Talking Face Generation · p. 2
Speaker-arbitrary Talking Face Generation
- The proposed pipeline combines reference-image keypoints and pose with audio features and phonemes to generate dense motion fields, which an image renderer turns into video.A temporal discriminator and a lip-sync discriminator are used during training, while relative motion transfer refines dense motions at inference.Source: Figure 2 · p. 3
- Speaker-arbitrary methods trained across multiple speakers can struggle to learn a consistent speaking style, producing unnatural mouth shapes.Some prior systems also show lip-sync artifacts or rely on poses taken from other videos rather than audio.Source: Speaker-arbitrary Talking Face… · pp. 2–3
Proposed Method
The framework learns audio-visual motion correlations from one speaker and transfers the resulting motion fields to a reference image. It combines a phoneme- and pose-conditioned transformer with sequence-level training and relative motion transfer.
Proposed Method
- The pipeline predicts head motion, extracts reference-image keypoints, maps audio to dense motion fields, and renders the resulting talking-face images.It uses a head motion predictor, keypoint detector, Audio-Visual Correlation Transformer, and image renderer.Source: Proposed Method · p. 3
Audio-visual Correlation Transformer (AVCT)
- AVCT uses phonemes and head poses in a temporal window, with reference-image keypoint features and acoustic features, to predict keypoints and Jacobians representing dense motion fields.The encoder models phoneme-pose correlations; acoustic features in the decoder capture audio energy changes relevant to mouth amplitude.Source: Audio-visual Correlation Transformer… · p. 3, Encoder · p. 3, Decoder · p. 4
Batched Sequential Training
- Batched Sequential Training generates successive frames in parallel and adds sequence-level constraints to improve temporal consistency.A temporal discriminator evaluates sequences, while a lip-sync discriminator supplies an additional synchronization constraint.Source: Batched Sequential Training · p. 4
Head Motion Predictor
- The head motion predictor is trained on the specific speaker using projected pose images, with a first-frame pose loss aligning its predicted sequence to the reference.This alignment supports the relative motion transfer used at inference.Source: Head Motion Predictor · pp. 4–5
Relative Motion Transfer
- At inference, relative motion transfer maps the predicted motion relative to its first frame onto keypoints and Jacobians detected from the reference image.The transferred motion fields are then used to render the output videos.Source: Relative Motion Transfer · p. 5
Experiments
Across HDTF and VoxCeleb2, the method performs best on most evaluation metrics. Qualitative comparisons, ablations, and a user study assess its mouth shapes, synchronization, head motions, and visual quality.
Experiments
- The experiments train on 20 hours of Obama video and evaluate on HDTF and VoxCeleb2 using FID, CPBD, mouth LMD, AVOff, and AVConf.Relative landmark distance is used to reduce the influence of head pose and image resolution.Source: Dataset · p. 5, Evaluation Metrics · pp. 5–6
Quantitative Evaluation
- The method achieves the best performance on most metrics across HDTF and VoxCeleb2; Wav2Lip has the highest HDTF CPBD score, although its mouth areas are blurry.The authors attribute Wav2Lip’s CPBD score to preserving most of the reference image.Source: Quantitative Evaluation · p. 6
Qualitative Evaluation
- Against one-shot methods, the method produces the most accurate mouth shapes and best preserves identity; comparisons also report authentic lip-sync and natural mouths.Wav2Lip and PC-AVS achieve similar lip-sync, but their mouth shapes look mechanical; Audio2Head has inferior lip-sync and FGTF produces distorted mouths.Source: Comparisons with Speaker-arbitrary… · pp. 6–7
- For Obama, the method produces videos comparable to speaker-specific methods and achieves accurate mouth shapes without SynObama’s sometimes-visible mouth-compositing artifacts.The comparison uses frames pronouncing the same phonemes; the method is end-to-end and requires no additional editing.Source: Figure 4 · p. 6, Comparisons with Speaker-specific… · p. 7
Ablation Study
- Ablations show poorer image quality when removing the reference image or replacing keypoints with it, and poorer temporal stability and lip-sync without Dseq and Dsync.Removing phonemes also prevents extension to unseen timbre and produces out-of-sync videos; audio features affect vivid mouth movements.Source: Ablation Study · p. 7
- The SynObama–FOMM combination can produce inferior results when its first-frame pose differs from the reference, while the proposed model preserves the initial pose.FOMM requires similar initial poses for the reference and one-shot video.Source: Figure 6 · p. 6, Ablation Study · p. 7
User Study
- In a study of 19 volunteers rating 21 videos, the method outperforms competitors in lip-sync quality, head-movement naturalness, and video realness.Participants rated each video from 1 to 5; Table 3 reports average scores across the videos.Source: User Study · p. 7, Table 3 · p. 7
Conclusion
The framework learns consistent audio-visual correlations from one speaker and transfers speaking styles to other subjects, producing realistic talking faces and animating non-photorealistic paintings.
Conclusion
- The framework learns consistent audio-visual correlations from a single speaker and transfers talking styles to different subjects, generating photo-realistic videos with accurate lip-sync, natural lip shapes, and rhythmic head motions.The authors also demonstrate talking-head animation for non-photorealistic paintings.Source: Conclusion · p. 7