Source-linked AI summary
Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation
Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, Ziwei Liu
TL;DR
Accurate lip synchronization has not resolved efficient head-pose control, especially without relying on potentially inaccurate structural estimates. PC-AVS uses an implicit low-dimensional pose code and complementary audio-visual reconstruction to control pose from another video, producing accurate lip synchronization with robustness to extreme views and frontalization.
Problem
Efficient head-pose control remains challenging because prior methods rely on structural representations and audio alone cannot provide absolute pose information.
Method
PC-AVS modularizes speech content, head pose, and identity using an implicit low-dimensional pose code learned from non-identity visual features with modulated convolutional reconstruction.
Results
The model generates accurately lip-synced talking faces whose poses are controllable by another video, with robustness to extreme views and talking-face frontalization.
Takeaways & Limitations
The framework provides pose-controllable talking-face generation from audio and a pose-source video without structural intermediate information.
Takeaways & Limitations
The formulation assumes audio and visual features share the same space.
Abstract
from arXiv · showhide
While accurate lip synchronization has been achieved for arbitrary-subject audio-driven talking face generation, the problem of how to efficiently drive the head pose remains. Previous methods rely on pre-estimated structural information such as landmarks and 3D parameters, aiming to generate personalized rhythmic movements. However, the inaccuracy of such estimated information under extreme conditions would lead to degradation problems. In this paper, we propose a clean yet effective framework to generate pose-controllable talking faces. We operate on raw face images, using only a single photo as an identity reference. The key is to modularize audio-visual representations by devising an implicit low-dimension pose code. Substantially, both speech content and head pose information lie in a joint non-identity embedding space. While speech content information can be defined by learning the intrinsic synchronization between audio-visual modalities, we identify that a pose code will be complementarily learned in a modulated convolution-based reconstruction framework. Extensive experiments show that our method generates accurately lip-synced talking faces whose poses are controllable by other videos. Moreover, our model has multiple advanced capabilities including extreme view robustness and talking face frontalization. Code, models, and demo videos are available at https://hangz-nju-cuhk.github.io/projects/PC-AVS.
1. Introduction
PC-AVS addresses pose control in audio-driven talking faces without structural intermediate representations by separating speech content, head pose, and identity through implicit latent codes. It uses cross-frame reconstruction with modulated convolution to generate lip-synced faces whose pose can be controlled by another video.
- Previous methods achieve accurate mouth movements but often fail to model head pose, which is important for natural-looking talking faces.
- Head-pose control is difficult because pose is rarely inferable from audio and prior controllable methods rely on short reference clips or 3D structural representations.
- PC-AVS uses another pose-source video to control head motions while driving an arbitrary face with audio, without requiring structural information.
- The method maps a non-identity visual embedding to a low-dimensional pose code and uses identity supervision to modularize speech content, head pose, and identity.
- Cross-frame reconstruction and modulated convolution encourage the pose code to control head pose while the speech-content embedding controls mouth motion.
- The system generates pose-controllable talking faces with accurate lip synchronization, little preprocessing, and robustness to input views.
2. Related Work
Related work for audio-driven talking faces includes structural and reconstruction-based approaches. Existing methods improve lip synchronization but commonly leave head pose unchanged or entangle pose with facial movements.
- Earlier talking-face systems used structural information and stitching, while later methods generated landmarks through temporal modeling but were often speaker-specific.
- Reconstruction-based methods can synchronize mouths with audio but normally neglect head movements because decoupling head pose from facial movements is difficult.
- 2D landmark methods entangle pose and mouth information, whereas 3D tools provide stronger intermediate representations for preserving compact information.
3. Our Approach
PC-AVS identifies a non-identity visual space and modularizes speech content, head pose, and identity into separate representations for pose-controllable talking-face generation. It learns an implicit low-dimensional pose code from this space and combines the representations through modulated-convolution reconstruction.
- Framework: PC-AVS drives a static identity reference with audio while using a separate pose source video, avoiding structural intermediate representations.The framework targets arbitrary talking faces and uses another video to provide head-motion information.
- Identifying Non-Identity Feature Space: Data augmentation identifies a non-identity space by varying color, perspective, and centered crops to reduce irrelevant texture and facial-structure effects.Perspective transformation uses source and target points with random margins and horizontal shifts; the transformation is represented by a 3 × 3 matrix M.
- Learning Speech Content Space: Synchronized audio-visual features define the speech content space, learned with contrastive matching between timely aligned pairs and negative audio examples.The audio-to-visual objective is trained symmetrically with the visual-to-audio objective, and their losses are summed.
- Devising Pose Code: The pose space maps non-identity features to a 12-dimensional code motivated by the dimensions of rotation, translation, and scale, without supervising real 3D parameters.The code is intentionally implicit and may not have the numerical values of traditional 3D pose parameters.
- Talking Face Generation: Target-frame reconstruction concatenates identity, speech-content, and pose features, then uses the resulting code to modulate convolutional generator weights.A multilayer perceptron maps the concatenated features to per-channel modulation vectors; joint reconstruction training complements the representation spaces.
4. Experiments
Experiments evaluate PC-AVS on VoxCeleb2 and LRW using objective metrics, user studies, qualitative comparisons, ablations, and extreme-view tests. The results indicate accurate lip synchronization, diverse pose motion, robustness to difficult views, and frontalization.
- Experimental Settings: The evaluation uses VoxCeleb2 and LRW, with VoxCeleb2 containing diverse quality and extreme-pose cases while LRW is mostly clean and frontal.The datasets differ substantially in pose difficulty, image quality, and whether test identities appear during training.
- Experimental Settings: PC-AVS is compared using SSIM, CPBD, mouth-region LMD, and Syncconf for generation quality, sharpness, mouth shape, and lip synchronization.LMD is lower-is-better; the other listed metrics are higher-is-better.
- Qualitative Evaluation: PC-AVS produces more diverse head motions and more accurate lip shapes than the compared methods in qualitative examples.ATVG fails on one difficult case, while MakeitTalk has less accurate mouth shapes and Rhythmic Head has worse identity preservation.
- User Study: Wav2Lip matches PC-AVS on lip synchronization, while users prefer PC-AVS for head-motion naturalness and video realness.ATVG and Wav2Lip generate near-stationary results, yielding lower head-movement and realness scores.
- Ablation Studies: Removing contrastive loss or target-frame augmentation disrupts synchronization and pose-code learning, while larger pose codes reduce both metrics under equal training time.The ablations vary contrastive loss, pose feature length, and generator structure on VoxCeleb2.
- Further Analysis: PC-AVS handles large poses and frontalizes faces while preserving identity when the pose code is set to zero.The extreme-view comparison reports artifacts or failures for competing methods in difficult inputs.
5. Conclusion
PC-AVS generates accurately lip-synced talking faces with pose freely controlled by another video, without structural intermediate information. Its modular representation also supports robustness under extreme poses and viewpoints.
- PC-AVS generates accurately lip-synced talking faces with pose freely controlled by another pose source video.
- The framework uses an implicit pose code and modularizes representations into identity, speech content, and pose spaces without structural intermediate information.
- Complementary learning improves lip synchronization relative to previous works.
- The model remains robust under extreme conditions, including large poses and viewpoints.