Source-linked AI summary
FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learning
Chenxu Zhang, Yifan Zhao, Yifei Huang, Ming Zeng, Saifeng Ni, Madhukar Budagavi, Xiaohu Guo
TL;DR
Talking-face synthesis must produce synchronized lip motion while also modeling weakly audio-correlated head poses and eye blinks. FACIAL-GAN jointly learns these attributes from phonetic, contextual, and identity information, and its rendering-to-video network produces photorealistic videos with natural facial motion that surpasses state-of-the-art results. The method focuses on head poses and eye blinks, while other implicit attributes remain future directions.
Problem
Photorealistic talking-face generation must synchronize lip motions while also producing natural, personalized head movements and eye blinks that are weakly correlated with audio.
Method
FACIAL-GAN jointly models explicit and implicit attributes using phonetic, contextual, and personalized information, while Rendering-to-Video uses rendered faces and eye-blink attention maps to generate frames.
Results
The end-to-end FACIAL framework generates photorealistic talking-face videos with synchronized lip motion, natural head motion, and realistic eye blinks, with better quality than state-of-the-art methods.
Takeaways & Limitations
Jointly learning explicit and implicit attributes enables talking-face videos with synchronized lips, personalized head motion, and realistic eye blinks.
Takeaways & Limitations
The work targets head poses and eye blinks; gaze motion, body and hand gestures, and microexpressions require further designs.
Abstract
from arXiv · showhide
In this paper, we propose a talking face generation method that takes an audio signal as input and a short target video clip as reference, and synthesizes a photo-realistic video of the target face with natural lip motions, head poses, and eye blinks that are in-sync with the input audio signal. We note that the synthetic face attributes include not only explicit ones such as lip motions that have high correlations with speech, but also implicit ones such as head poses and eye blinks that have only weak correlation with the input audio. To model such complicated relationships among different face attributes with input audio, we propose a FACe Implicit Attribute Learning Generative Adversarial Network (FACIAL-GAN), which integrates the phonetics-aware, context-aware, and identity-aware information to synthesize the 3D face animation with realistic motions of lips, head poses, and eye blinks. Then, our Rendering-to-Video network takes the rendered face images and the attention map of eye blinks as input to generate the photo-realistic output video frames. Experimental results and user studies show our method can generate realistic talking face videos with not only synchronized lip motions, but also natural head movements and eye blinks, with better qualities than the results of state-of-the-art methods.
1. Introduction
Talking-face generation must synchronize strongly audio-linked lip motions while also producing weakly audio-correlated, personalized head movements and eye blinks. FACIAL addresses this gap by jointly learning explicit and implicit attributes with phonetic, contextual, and identity information.
- Photorealistic talking faces require synchronized lips plus natural, personalized head movements and eye blinks, which remain challenging to generate together.
- Existing methods mainly synchronize lip motions with audio, while only a few explore audio-related implicit attributes such as head pose.
- FACIAL jointly learns explicit and implicit attributes, embedding eye-blink AU, head pose, expression, identity, texture, and lighting within one adversarial framework.
- FACIAL-GAN integrates phonetic, contextual, and personalized information by combining sequence-level context with frame-level phonetic encoding.
- The framework predicts eye-blink action units and embeds them into an eye-attention map for realistic blinking in the rendered video.
- The end-to-end FACIAL framework produces photorealistic talking faces with synchronized lips, natural personalized head motion, and realistic eye blinks, outperforming state-of-the-art methods.
2. Related Work
Prior audio-driven methods largely focus on lip synchronization and often produce nearly fixed head poses. FACIAL combines learned phonetic, contextual, and personalized information with 3D modeling and eye-blink attention to address these limitations.
- Most audio-driven talking-face methods synchronize facial motion with audio but generate videos with nearly fixed head poses.
- The framework overview combines explicit expression, implicit eye blinking and head pose, reconstructed 3D guidance, and eye attention before rendering-to-video.
- FACIAL-GAN integrates phonetic, contextual, and personalized information, then combines a synthesized 3D model with AU attention to generate talking-face videos.
- Video-driven methods transfer expressions and limited head movements from source videos, whereas FACIAL targets audio-driven synthesis with personalized head poses and eye blinks.
3. Approach
The approach combines FACIAL-GAN, which jointly models explicit and implicit facial attributes, with a Rendering-to-Video network that produces realistic output frames. FACIAL-GAN uses temporal and local audio features, while eye attention maps help render personalized head motion and blinking.
- 3.1. Problem Formulation: FACIAL-GAN jointly encodes explicit and implicit attributes, addressing the conventional pipeline’s inability to capture dynamic head poses and eye blinks.The framework combines adversarial attribute learning with an auxiliary eye-region attention map.
- 3.2. FACIAL-GAN: The temporal generator processes the whole T-frame audio sequence, while the local phonetic generator extracts frame-specific features before predicting facial expression, head pose, and blink parameters.Gtem produces contextual latent features, and Gloc uses a local audio window for each frame.
- 3.2. FACIAL-GAN: The generator is supervised with attribute, initial-state, and motion losses, while a facial discriminator adversarially distinguishes real from generated attributes.The motion loss encourages continuity between adjacent predicted frames, and the initial-state loss supports continuity across sliding windows.
- 3.3. Rendering and Eye Attention: The method combines reference-video geometry, texture, and illumination with generated expression and pose coefficients to render a personalized 3D face.Because 3D reconstruction may miss subtle eye motion, the method applies blink values through an eye attention map restricted to the eye region.
- 3.4. Rendering-to-Video Network: The Rendering-to-Video network consumes stacked rendered frames and eye attention maps, using adversarial, feature-matching, perceptual, and pixel losses to generate photorealistic target-person frames.A temporal window is used to promote coherency across output frames.
4. Dataset Collection
The dataset combines audio, 3D face, head-pose, eye-motion, and identity-related information to support joint modeling of explicit and implicit talking-face attributes.
- The dataset is designed to combine explicit and implicit attributes, addressing prior datasets with fixed poses, short heterogeneous clips, or limited person-specific information.
- Audio features are extracted with DeepSpeech at 50 FPS, then linearly resampled to 30 FPS to match the video frames.
- OpenFace provides six-dimensional rigid head pose and eye-region Action Units that represent motion intensities around the eyes.
- The 3D face representation includes geometry, expression, texture, and spherical-harmonics illumination coefficients from a 3D Morphable Model.
- The dataset contains more than 450 approximately one-minute clips, totaling 535,400 frames at 30 FPS with a 5-1-4 train-validation-test split.
5. Experiments
Experiments compare the proposed system with established talking-face and facial-reenactment methods, quantify attribute quality, test architectural components, and assess human judgments.
- 5.2.1 Qualitative Comparison: Compared with ATVG, DAVS, Zhou et al., and Yi et al., the method produces personalized head movements and realistic eye blinking rather than static or subtle motions.
- 5.2.1 Qualitative Comparison: Facial-reenactment baselines generate photorealistic videos but retain implicit attributes from the source video, limiting generated-video length to the reference video without special connections.
- 5.2.2 Quantitative Evaluation: The proposed model surpasses most existing methods quantitatively and achieves higher visual quality than the compared methods on the same characters.
- 5.3. Ablation Studies: Removing either temporal or local phonetic features reduces the reported SyncNet confidence from 5.127 to 4.051 or 4.309, respectively.
- 5.3. Ablation Studies: Without eye-attention maps, blinking frequencies are extremely low and unnatural, whereas the full method produces blink frequency and duration distributions similar to real videos.
- 5.4. User Studies: In a user study with 20 volunteers, the proposed method was rated better than state-of-the-art methods across photo-realism, lip synchronization, head motion, and eye blinking.
6. Discussion and Future Work
The work focuses on implicit attribute learning for natural head poses and eye blinks, while recognizing that other talking-face attributes require further exploration.
- 6. Discussion and Future Work: The framework targets natural head poses and eye blinks, leaving gaze motion, body and hand gestures, and microexpressions for future work.These attributes may be guided by other information dimensions and require specific network components.