Source-linked AI summary
Progressive Disentangled Representation Learning for Fine-Grained Controllable Talking Head Synthesis
Duomin Wang, Yu Deng, Zixin Yin, Heung-Yeung Shum, Baoyuan Wang
TL;DR
Talking-head synthesis methods often lack precise, separate control over facial motions, while lip movement and emotion can be entangled. PD-FGC progressively disentangles latent motion representations from in-the-wild video data. It provides precise control over diverse facial motions, reports best motion-control accuracy on NLSE-C, LMD, and LMDm, and shows competitive image quality.
Problem
Existing methods may not support precise individual control of facial factors, which are difficult to disentangle because motions such as lip movement and emotion are entangled.
Source: §1 · p. 2Method
PD-FGC progressively separates motion factors from a unified feature, using motion-specific learning for non-emotional motions and feature-level decorrelation with self-reconstruction for expression.
Source: §1 · p. 2, §3.3 · p. 5Results
The method provides precise control over diverse facial motions, achieves the best motion-control accuracy on NLSE-C, LMD, and LMDm, and has competitive image generation quality.
Source: §1 · p. 2, §4.1 · p. 6Takeaways & Limitations
Experiments demonstrate efficacy for disentangled, fine-grained control of diverse facial motions.
Source: §5 · p. 8Synthesized images may lack fine details, which the authors leave for future work.
Source: §5 · p. 8
Abstract
from arXiv · showhide
We present a novel one-shot talking head synthesis method that achieves disentangled and fine-grained control over lip motion, eye gaze&blink, head pose, and emotional expression. We represent different motions via disentangled latent representations and leverage an image generator to synthesize talking heads from them. To effectively disentangle each motion factor, we propose a progressive disentangled representation learning strategy by separating the factors in a coarse-to-fine manner, where we first extract unified motion feature from the driving signal, and then isolate each fine-grained motion from the unified feature. We introduce motion-specific contrastive learning and regressing for non-emotional motions, and feature-level decorrelation and self-reconstruction for emotional expression, to fully utilize the inherent properties of each motion factor in unstructured video data to achieve disentanglement. Experiments show that our method provides high quality speech&lip-motion synchronization along with precise and disentangled control over multiple extra facial motions, which can hardly be achieved by previous methods.
1. Introduction
Fine-grained control of facial motions is difficult because motions can be entangled and existing methods may not support precise individual control. PD-FGC progressively separates motion factors from a unified motion feature, using motion-specific learning for non-emotional motions and decorrelation plus self-reconstruction for expression.
1. Introduction
- Existing methods may not modify eye gaze or expression independently, or may struggle to control individual facial factors precisely.Lip motion is also highly entangled with emotion, and large-scale supervised annotations for disentangling these factors are insufficient.Source: §1 · p. 2
- PD-FGC targets one-shot synthesis with separate control of lip motion, head pose, eye gaze and blink, and emotional expression.Lip motion is driven by audio; other motions can be driven individually by videos.Source: §1 · p. 2
- Appearance and motion are first disentangled through augmentation and self-driving to produce a unified motion feature that excludes appearance.This feature retains the driving frame’s motions and serves as the starting point for finer disentanglement.Source: §1 · p. 2
- Lip motion, eye gaze and blink, and head pose are separated from the unified feature using motion-specific contrastive learning and 3D pose guidance.The method uses shared information between audio and unified motion features for speech-only lip motion, eye-region contrastive learning for eye motion, and 3D rigid transformation for head pose.Source: §1 · p. 2
- Expression is disentangled through feature-level decorrelation from other motions, alongside self-reconstruction to learn a semantically meaningful expression representation.The paper defines emotional expression as facial expression excluding speech-related mouth movement and eye gaze and blink.Source: §1 · p. 2
2. Related work
Related work spans audio-driven synthesis, video-driven face reenactment, and disentangled face representation learning. PD-FGC combines audio control of lip motion with progressive learning of separate motion representations.
Audio-driven talking head synthesis
- Audio-driven methods prioritize accurate lip motion from speech [9] [46], while many control only the mouth [9] [39] [46] [79]; newer methods add facial properties or expression [8] [70] [28] [29].Some expression methods use manually collected datasets [28] [29], while GC-AVT uses in-the-wild data; PD-FGC instead uses feature-level decorrelation and controls lip motion, head pose, eye gaze and blink, and expression individually.Source: §2 · pp. 2–3
Video-driven face reenactment
Disentangled representation learning on the face
- Face disentanglement work includes unsupervised methods [11] [12] [18] [33] [41], GAN latent editing [80], and prior-guided learning [16] [19] [22].The paper notes limitations in meaningful alignment, classifier- or generator-constrained control, and structured-data reliance; PD-FGC uses progressive training on videos with prior models.Source: §2 · p. 3
3. Method
The method progressively separates appearance and motion, then disentangles lip motion, eye gaze and blink, head pose, and expression. It combines motion-specific learning with expression decorrelation and self-reconstruction to train a controllable talking-head generator.
3. Method
- The method first extracts a unified motion representation, then isolates fine-grained motions, and finally separates expression while learning an image generator.The resulting generator synthesizes talking heads from disentangled motion and appearance features.Source: §3 · p. 3, Figure 2 · p. 4, §3.3 · p. 6
3.1. Appearance and Motion Disentanglement
- Appearance and motion encoders, self-driving, reconstruction, and motion-branch augmentation produce a unified motion feature that represents facial movements while excluding appearance.The motion reconstruction loss compares features from a synthesized image and the ground-truth image.Source: §3.1 · pp. 3–4
3.2. Fine-Grained Motion Disentanglement
- Lip, eye, and head-pose features are isolated from a unified motion feature using motion-specific contrastive learning and 3D pose guidance.Audio–visual contrastive learning aligns lip features with corresponding audio; eye-region contrastive learning focuses on eye motion; a 3D prior guides head-pose regression.Source: §3.2 · pp. 4–5
3.3. Expression Disentanglement
- Expression features are decorrelated from other motions and combined with appearance and motion features to self-reconstruct driving frames, while training the final controllable image generator.In-window averaging and expression–audio decorrelation address motion entanglement; self-reconstruction encourages the expression feature to carry complementary semantic information.Source: §3.3 · pp. 5–6
4. Experiments
Experiments evaluate PD-FGC quantitatively, qualitatively, and through a user study. Results support fine-grained motion control, with reported advantages in motion-control accuracy and user-rated expression control and naturalness.
4. Experiments
- Experiments train on VoxCeleb2 and evaluate on VoxCeleb2 and Mead, using self-driving and cross-video settings and normalized NLSE-C alongside landmark metrics.NLSE-C measures generated-image confidence relative to the training data and is reported to align better with human perception.Source: §4 · p. 6, §4.1 · pp. 6–7
4.1. Quantitative Evaluation
- PD-FGC achieves the best NLSE-C, LMD, and LMDm motion-control accuracy, and the lowest expression and pose control errors against prior methods.Image-generation quality is reported as competitive; Wav2Lip is excluded from FID and LMD evaluation because copying other regions could make those comparisons unfair.Source: §4.1 · pp. 6–7, Table 1 · p. 7, Table 2 · p. 7
4.2. Qualitative Evaluation
- The method independently controls each motion to mimic its driving source while leaving other properties unchanged, or sets other motion factors to canonical zero positions.Figure 3 illustrates both control modes.Source: §4.2 · p. 7, Figure 3 · p. 7
- In visual comparisons, PC-AVS and EAMM show limitations in controlling eye motion, head pose, or expression relative to the driving source.PC-AVS controls head pose and lip motion but has different eye motions or expressions and inferior head-pose control; EAMM does not control head pose and eye motions well and produces expressions unlike the driving source.Source: §4.2 · pp. 7–8, Figure 4 · p. 8
4.3. User Study
- User-study participants rated PD-FGC best for expression control quality and facial-motion naturalness, and second-best for lip-motion synchronization, close to PC-AVS.Participants scored synthesized-image properties from 1 to 5.Source: Table 3 · p. 7, §4.3 · p. 8
4.4. Ablation Study
- Both feature-level decorrelation strategies improve expression and lip-motion disentanglement with only a slight reduction in expression-control accuracy, while window size 13 gives the best in-window decorrelation result.Using both decorrelation strategies yields the best ablation result.Source: Table 4 · p. 8, Table 5 · p. 8, §4.4 · p. 8
5. Conclusion
The method separates facial motions progressively into disentangled latent representations, and experiments demonstrate fine-grained control; synthesized images may lack fine details.
5. Conclusion
- The method uses progressive disentangled representation learning to separate individual facial motion factors in a coarse-to-fine manner.Experiments demonstrate disentangled, fine-grained control of diverse facial motions.Source: §5 · p. 8
- The conclusion presents experiments as demonstrating the efficacy of the method for fine-grained control of diverse facial motions.Source: §5 · p. 8
- Synthesized images may lack fine details, which the authors leave for future work.Source: §5 · p. 8
A.1. Data Preparation
The appendix describes training and evaluation video sources, frame preparation, and audio preprocessing.
A.1. Data Preparation
- Training uses all available VoxCeleb2 training-split videos; evaluation samples 500 VoxCeleb2 and 460 MEAD test clips.Source: §A.1 · p. 13
- Frames are aligned using official annotations, then resized and center-cropped to 224 × 224.Source: §A.1 · p. 13
- Audio is extracted with ffmpeg, sampled at 16,000, and converted to 80-band Mel-spectrograms using a 1,280-sample window and 160-sample hop.Source: §A.1 · p. 13
A.2. More Training Details
The appendix details separate training stages for appearance and motion, lip, eye, pose, and expression representations.
A.2. More Training Details
- Appearance and motion encoders are trained with a single appearance reference, self-driving, and an added motion-reconstruction loss.The stage trains for 50 epochs with batch size 24 on eight Tesla V100 GPUs, taking around two weeks.Source: §A.2 · p. 13
- Lip and audio encoders use audio-visual contrastive learning, with unsynchronized frames and audio from the same clip as negative pairs.The networks train for 30 epochs with batch size 32, one positive pair and eight negative pairs per item.Source: §A.2 · p. 13
- Eye-motion contrastive learning is motivated by replacing the driving frame’s eye region, which permits eye control without changing lip motion in the illustrated scenario.Eye and pose encoders each train for 30 epochs with batch size 128.Source: §A.2 · p. 13, Figure I · p. 13
- Expression encoding and the final generator use feature-level decorrelation and complementary self-reconstruction while the other networks remain fixed.The expression encoder is trained for the first 40,000 iterations, then frozen; in-window decorrelation uses a window size of 13 and lip-motion decorrelation a memory bank of 512.Source: §A.2 · p. 14
A.3. Quantitative Evaluation Details
The appendix describes two evaluation settings, user studies, qualitative synthesis, motion-factor controllability, and expression interpolation.
A.3. Quantitative Evaluation Details
- The evaluation uses self-driving clips for ground-truth comparison and cross-video clips to assess expression transfer between source and target identities.In cross-video evaluation, expression parameters are extracted from synthesized videos and driving frames and compared.Source: §A.3 · p. 14
A.4. User Study Details
- One user study scores lip synchronization, expression control, and naturalness; another rates variation when only one motion factor is driven.The first uses 35 videos and 15 participants; the second reports results in Table VII.Source: §A.4 · p. 14
B.1. Fine-Grained Controllable Talking Heads
- Examples show the method combining motions from different driving sources to synthesize talking heads.Source: §B.1 · p. 14
B.2. Disentangled Controllability
- Variance for each motion factor is highest when that same factor is controlled and decreases under other controls, indicating minor cross-motion influence.Source: §B.2 · p. 15, Table VI · p. 15
- Some computed variances remain large because off-the-shelf feature extractors can respond to other motions; the authors also point to video inspection and user-study results.In the user study, scores are close to 5 when the measured and controlled factors match, and close to 1 when they differ.Source: §B.2 · p. 15, Table VII · p. 15
B.3. Expression Interpolation
- Interpolating between expression features from different sources produces smooth transitions and natural expressions, supporting continuous expression control.Source: §B.3 · p. 15
B.4. Comparison with the prior methods
In the self-driving setting, the method produces lip motion described as natural and closer to ground truth than the alternatives.
B.4. Comparison with the prior methods
- The synthesized lip motion is natural and closer to ground truth than the alternatives.The comparison uses images synthesized with lip motion driven by the audio signal.Source: §B.4 · p. 15
- The comparison evaluates lip motion under the self-driving setting, with audio driving the motion.Source: §B.4 · p. 15
- The paper directs readers to accompanying video animations for the comparison.Source: §B.4 · p. 15
B.5. Ablation Study
The ablation finds motion reconstruction loss important for informative unified motion features and accurate control. The ethics discussion warns that the method could be misused to create deceptive content, including fake celebrity videos.
B.5. Ablation Study
- With motion reconstruction loss, synthesized facial motions contain more detail and are closer to driving sources; without it, rich-expression inputs yield poor reenactment.The authors say the loss helps produce an informative unified motion feature for accurate control.Source: §B.5 · p. 15
C. Ethics Consideration
- The authors warn that the method could be misused to create deceptive, harmful content, especially fake videos of celebrities.They suggest investigating forgery detection to prevent illegal use.Source: C. Ethics Consideration · p. 16
B.5. Ablation Study
- Figure VI ablates motion reconstruction loss in appearance-and-motion disentanglement learning.Source: Figure VI · p. 18