Source-linked AI summary
Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion
Suzhen Wang, Lincheng Li, Yu Ding, Changjie Fan, Xin Yu
TL;DR
One-shot talking-head generation must produce natural audio-matched head motion while preserving speaker appearance and stabilizing non-face regions during large movements. Audio2Head separates rigid 6D head-pose prediction from detailed motion-field generation and image rendering, achieving high-fidelity videos with natural rhythmic motion, few artifacts, and competitive synchronization, while retaining known limitations in lip-sync, blinking, and extreme references.
Problem
Existing one-shot methods often neglect audio-driven head motion, while head-motion methods face ambiguous audio-motion correspondences and distort non-face regions.
Method
The framework predicts rigid 6D head poses with a motion-aware RNN, generates integrated keypoint-based dense motion fields from audio, poses, and a reference image, then renders video frames.
Results
The method produces high-fidelity videos with natural-looking rhythmic head motions, few non-face and temporal artifacts during large pose changes, and competitive lip-sync accuracy.
Takeaways & Limitations
Integrating facial, head, and background motion enables realistic full-background talking-head videos while preserving speaker identity through large pose changes.
Takeaways & Limitations
Lip-sync accuracy drops for bilabial and labiodental phonemes, blinking is not captured, and extreme-pose or extreme-expression references remain difficult.
Abstract
from arXiv · showhide
We propose an audio-driven talking-head method to generate photo-realistic talking-head videos from a single reference image. In this work, we tackle two key challenges: (i) producing natural head motions that match speech prosody, and (ii) maintaining the appearance of a speaker in a large head motion while stabilizing the non-face regions. We first design a head pose predictor by modeling rigid 6D head movements with a motion-aware recurrent neural network (RNN). In this way, the predicted head poses act as the low-frequency holistic movements of a talking head, thus allowing our latter network to focus on detailed facial movement generation. To depict the entire image motions arising from audio, we exploit a keypoint based dense motion field representation. Then, we develop a motion field generator to produce the dense motion fields from input audio, head poses, and a reference image. As this keypoint based representation models the motions of facial regions, head, and backgrounds integrally, our method can better constrain the spatial and temporal consistency of the generated videos. Finally, an image generation network is employed to render photo-realistic talking-head videos from the estimated keypoint based motion fields and the input reference image. Extensive experiments demonstrate that our method produces videos with plausible head motions, synchronized facial expressions, and stable backgrounds and outperforms the state-of-the-art.
1 Introduction
Audio2Head addresses natural audio-matched head motion and artifact-free one-shot talking-head generation by separating holistic head movement from detailed facial motion and modeling full-image motion.
- Method: The image-generation network renders photo-realistic talking-head frames from the estimated motion fields and the single reference image.Figure 1 depicts the sequence from reference image and audio, through predicted pose and motion fields, to synthesized frames.
- Motivation: Most one-shot methods synchronize lip movements but neglect audio-driven head motion, while existing head-motion methods face ambiguous training correspondences and background distortions.Different performers can realize identical audio with opposite head movements, encouraging still-like predictions; face-focused methods also distort non-face regions.
- Method: The framework predicts rigid 6D head motions with a motion-aware RNN as low-frequency holistic movement, allowing another network to generate detailed facial movements.The method disentangles head motions from expression changes to address ambiguity in audio-motion correspondence.
- Method: A keypoint-based dense motion-field generator uses audio, predicted head poses, and a reference image to represent facial, head, and background movements integrally.This representation is intended to govern spatial and temporal consistency across the generated video.
- Results: Extensive comparisons report state-of-the-art visual quality and rhythmic head motions without noticeable artifacts.The paper presents this as its principal result for audio-driven single-image talking-head generation.
2 Related Work
Prior work includes one-shot audio-driven talking-head generation, dense-motion video driving, and head-motion prediction, while Audio2Head targets full-frame motion and more natural motion patterns.
- Audio-driven talking-head generation: Audio-driven one-shot methods reduce speaker reference information to a single image, while speaker-specific approaches require minutes to hours of training video for each speaker.Related methods use temporal modeling, GANs, or end-to-end audio-visual generation for talking-face videos.
- Video-driven methods: Dense-motion video-driven methods use self-learned keypoints to warp reference-image features for reenactment.The cited keypoint representation motivates Audio2Head’s use of dense motion fields.
- Head motion prediction: Traditional avatar methods model 3D head rotation, whereas 2D-image methods must model both head rotation and translation.Prior statistical constraints on pose mean and standard deviation cannot model local motion patterns, according to the cited discussion.
3 Proposed Method
The proposed method combines a head motion predictor with keypoint-based motion-field generation and image rendering to synthesize talking-head videos from reference images and audio.
- Overview: The pipeline takes a reference image and audio clip, then synthesizes synchronized video frames through four components.These components are head motion prediction, motion-field generation, keypoint detection, and image generation.
- Head Motion Predictor: The head motion predictor generates natural, rhythmic head-pose sequences representing low-frequency holistic movement.It extracts an initial head and body state from the reference image and uses a two-layer LSTM driven by acoustic features.
- Image Generation: The image generator renders synthesized images from relative dense motion between predicted keypoints and the reference image’s initial keypoints.Relative motion is defined through the predicted and reference keypoints, while multiresolution feature reconstruction contributes to training.
- Head Motion Predictor: The predictor uses SSIM loss on successive 6D pose vectors and an adversarial temporal discriminator to improve motion structure, fidelity, and smoothness.The head-motion sequence is treated as a 6 × T image for structural comparison, while PatchGAN applies 1D convolutions along temporal trunks.
- Motion Field Generator: The motion-field generator uses audio, reference-image identity, and predicted poses to produce keypoints and Jacobians controlling dense motion fields.The inputs are unified into tensors, including repeated reference-image features and projected pose representations.
- Motion Field Generator: The motion-field generator is trained in two stages, first guided by a pretrained keypoint detector and then fine-tuned with a pretrained image generator.The first stage uses keypoint, Jacobian, and heatmap losses; the second reconstructs frames for reconstruction-based fine-tuning.
4 Experiment Setup
The experiments use VoxCeleb, GRID, and LRW, with standardized video preprocessing and train/test splits. Figure 7 compares head-motion patterns across two reference images.
- Datasets: VoxCeleb, GRID, and LRW are used to evaluate the proposed method.VoxCeleb contains YouTube speech clips, GRID contains clips from 33 speakers, and LRW contains 500 words spoken by hundreds of people.
- Preprocessing: VoxCeleb and GRID videos are re-cropped and resized to 256×256, while all videos are sampled at 25 fps.
- Motion Comparison: Figure 7 compares head-motion patterns on two reference images, with bottom frames sampled from blue-box regions.
- Data Splits: Each dataset is split into training and testing sets following previous works.
- Implementation: The networks are implemented in PyTorch and trained with Adam using an initial learning rate of 2e-4 and weight decay of 2e-6.Different network groups use separate training schedules and batch sizes.
5 Experiments Results
The method produces higher visual quality, plausible head motions, competitive lip synchronization, and stable backgrounds, while ablations identify the contributions of its motion, Jacobian, loss, and refinement components.
- Visual Quality: The method generates more plausible head movements, more stable backgrounds, and fewer artifacts than compared audio-driven methods.It also preserves speaker identity after large pose changes and generalizes to unseen identities, paintings, and human-like statues.
- Quantitative Evaluation: The method achieves the highest PSNR/SSIM and lowest FID on GRID and VoxCeleb among the compared methods.LRW is different because its short videos contain little head-pose or background motion, favoring mouth-region editing.
- Audio-Visual Synchronization: The method obtains competitive lip-sync accuracy with SyncNet despite addressing the more challenging task of generating head motion and full-background videos.
- Head Motion: The head-motion predictor produces natural, rhythmic motions conditioned on both audio and identity, whereas MakeItTalk shows limited orientation changes and repetitive behaviors.The comparison uses the same audio with two reference images and visualizes six-dimensional motion through PCA.
- Jacobian Ablation: Removing Jacobians causes lips to exhibit mainly open-and-close patterns, indicating that local affine transformations benefit lip-shape details.
- Head-Pose Ablation: Replacing SSIM loss with L1 produces unnatural head-motion sequences, while removing SET reduces dynamics and audio synchronization.
- Training Refinement: Without second-stage refinement, generated videos contain texture inconsistency and slight jitters; pixel-level constraints improve temporal coherence and fidelity.
- User Study: In a 33-participant user study, 66.7% of cases were judged natural, significantly outperforming all compared methods.
6 Conclusion and Discussion
The framework combines separate head-motion prediction with dense motion-field generation and image rendering to produce high-fidelity videos with natural motion and few artifacts. Its main limitations are reduced lip-sync accuracy for some phonemes, missed blink patterns, and failures on extreme reference poses or expressions.
- Conclusion: The framework predicts head motions separately, generates keypoints for dense motion fields, and renders video frames from those fields.
- Conclusion: The method produces natural, rhythmic head motions, few non-face and inter-frame artifacts, and higher visual quality than state-of-the-art methods.
- Limitations: Lip-sync accuracy drops on bilabial and labiodental phonemes including p, f, and m.The paper characterizes this as a tradeoff for better head motion and visual quality.
- Limitations: The method cannot capture blink patterns and fails on reference images with extreme poses or expressions.
Ethical Impact
The authors identify applications such as video conferencing and movie dubbing, while warning that the method may be misused to create synthetic videos.
- Applications: The method may support applications including video conferencing and movie dubbing.
- Responsible Use: The authors plan to release code and models to promote fake-video detection and require outputs created with them to be marked synthetic.