Source-linked AI summary
One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing
Ting-Chun Wang, Arun Mallya, Ming-Yu Liu
TL;DR
The paper addresses one-shot talking-head synthesis with source-image appearance and driving-video motion, especially the fixed-viewpoint limitation of existing 2D methods. It learns an unsupervised decomposed 3D keypoint representation for free-view synthesis and compression, reporting better benchmark quality and 10x lower conferencing bandwidth than H.264 at comparable visual quality. Large occlusions and image degradation remain failure cases.
Problem
Existing 2D one-shot talking-head methods can synthesize only from the original viewpoint, limiting novel-view rendering.
Method
The method jointly learns an unsupervised 3D keypoint representation that decomposes facial expressions from a person’s geometric signature and combines them with head pose.
Results
The approach achieves better visual quality than state-of-the-art methods on benchmark datasets, supports local free-view control, and reduces video-conferencing bandwidth by 10x versus H.264 without compromising visual quality.
Takeaways & Limitations
Transmitting only keypoint transformations enables compact live-video streaming while modifying transformations supports more immersive free-view conferencing.
Takeaways & Limitations
The model fails under large occlusions and image degradation and does not guarantee pixel-aligned output videos.
Abstract
from arXiv · showhide
We propose a neural talking-head video synthesis model and demonstrate its application to video conferencing. Our model learns to synthesize a talking-head video using a source image containing the target person's appearance and a driving video that dictates the motion in the output. Our motion is encoded based on a novel keypoint representation, where the identity-specific and motion-related information is decomposed unsupervisedly. Extensive experimental validation shows that our model outperforms competing methods on benchmark datasets. Moreover, our compact keypoint representation enables a video conferencing system that achieves the same visual quality as the commercial H.264 standard while only using one-tenth of the bandwidth. Besides, we show our keypoint representation allows the user to rotate the head during synthesis, which is useful for simulating face-to-face video conferencing experiences.
1. Introduction
The paper introduces a one-shot neural talking-head synthesis approach that uses a source image for appearance and a driving video for motion, overcoming fixed-viewpoint limitations. It also supports local free-view control and substantially reduces video-conferencing bandwidth.
- Problem: The task generates a target person’s talking-head video from one source image and a driving video that supplies the output motions.The source image provides identity and appearance, while the driving video may come from another person.
- Problem: 2D one-shot methods are limited to synthesizing talking-heads from the original viewpoint and cannot render novel views.This limitation arises from the absence of 3D graphics models.
- Approach: The proposed method learns an unsupervised 3D keypoint representation that decomposes person-specific and motion-related information.The decomposition enables 3D transformations of the person-specific representation to simulate head-pose changes.
- Capabilities: The approach achieves local free-view synthesis, allowing the talking-head viewpoint to change within a large neighborhood of the original viewpoint.Head rotations can be synthesized in the output video by modifying the decomposed representation.
- Video conferencing: 10x bandwidth reduction is achieved versus commercial H.264 without compromising visual quality in video conferencing.The system transmits only the keypoint representation and reconstructs the source video on the receiver side.
2. Related Works
The related work spans GAN-based video synthesis, 3D and 2D talking-head methods, and controllable face generation. The paper positions its keypoint representation as enabling efficient compression while preserving facial movements, but not guaranteeing pixel-aligned output.
- GANs: GANs have been applied to image and video synthesis, and this work focuses on GANs for talking-head video synthesis.The cited applications include unconditional image synthesis, image translation, image processing, and video synthesis.
- 3D model-based talking-head synthesis: 3D model-based methods can synthesize inner-face regions well but struggle with realistic hair, teeth, and accessories, motivating modern 2D approaches.The passage states that most modern face-reenactment frameworks adopt the 2D approach because of these limitations.
- 2D-based talking-head synthesis: 2D talking-head methods include subject-dependent models limited to specific people and subject-agnostic models requiring a single image.The passage distinguishes the two categories by their training and input requirements.
- Compression and representation: The proposed representation exploits facial structure and relative part arrangement to support compression an order of magnitude better than traditional codecs.The method models facial movements and emotions but does not guarantee pixel-aligned output videos.
3. Method
The method decomposes unsupervisedly learned 3D keypoints into identity-specific geometry and motion-related pose and expression, then uses them to warp source features and generate video. This representation supports manual head-pose changes and compact transmission.
- The task generates output frames whose identity comes from one source image while motions come from a driving video.
- Source image feature extraction: The unsupervised keypoint system separates geometry signature, head pose, and facial expression into manipulable representations.
- Source image feature extraction: The model extracts a 3D appearance feature volume, enabling keypoint operations in 3D for rotating and translating the synthesized head.
- Source image feature extraction: Image-specific keypoints combine canonical points with rotation, translation, and expression deformation as xs,k = Rsxc,k + ts + δs,k.
- Video generation: The generator warps the 3D source feature using keypoint-induced flows and a learned composition mask before producing the output image.
- Driving video feature extraction: For each driving frame, the model reuses the source canonical keypoints and combines them with driving pose and expression to obtain xd,k = Rdxc,k + td + δd,k.
- User-specified rotation and translation can alter the synthesized head pose, while each driving frame is compactly represented for low-bandwidth conferencing.
4. Experiments
Experiments evaluate reconstruction, qualitative motion capture, cross-identity transfer, frontalization, and failure cases across VoxCeleb2 and TalkingHead-1KH. The reported comparisons favor the proposed method in motion fidelity, image quality, identity preservation, and frontalization realism.
- Experimental setup: The evaluation uses VoxCeleb2 and the newly collected TalkingHead-1KH dataset containing 180K talking-head videos.
- Experimental setup: The synthesis benchmarks measure reconstruction faithfulness, visual quality, and semantic consistency against FOMM, fs-vid2vid, and bi-layer neural avatars.
- Talking-head image synthesis: Qualitative VoxCeleb2 comparisons show that the method better captures driving motions.
- Talking-head image synthesis: On TalkingHead-1KH, the method produces more faithful and sharper results.
- Cross-identity motion transfer: In cross-identity motion transfer, the method achieves the best results compared to other methods, with lowest FIDs and highest identity-preserving scores.
- Failure cases: The model fails under large occlusions and image degradation.
- Face redirection: Only the proposed method is reported to realistically frontalize inputs, while pSp loses identity, RaR has inner-face problems, and both baselines have temporal-stability issues.
5. Neural Talking-Head Video Conferencing
The model turns talking-head video conferencing into transmission of compact motion information, allowing the receiver to reconstruct the sender’s face and adjust head pose. Its adaptive keypoint scheme achieves substantially lower bandwidth than H.264 while retaining comparable human-rated quality.
- Compression framework: 3K + 6 numbers compactly represent each driving image: 3K expression deformations, 3 rotation values, and 3 translation values.The values are entropy-encoded before transmission.
- Compression framework: The receiver decodes keypoint deformations and head pose, combines them with a source image, and reconstructs the driving image.The source image is transmitted once at session start or reused, rather than sent repeatedly like traditional I-frames.
- Adaptive transmission: Adaptive keypoint dropout dynamically reduces the number of transmitted keypoints to accommodate changing bandwidth requirements without compromising visual quality.The transmitted bit count is proportional to the number of keypoints.
- Evaluation: The video-conferencing evaluation uses 222 high-quality talking-head videos at 512x512 resolution, each up to 1024 frames long.H.264 is evaluated without bidirectional B-frames to match real-time streaming conditions.
- Evaluation: 10.37x and 6.5x bandwidth reductions are achieved against H.264 at CRF 36 using the adaptive and 20-keypoint schemes, respectively.Human preference scores indicate comparable quality between the adaptive method and H.264 at CRF 36.
- Evaluation: The method obtains better quality at much lower bandwidth than other neural talking-head methods because it transmits pose and deformations instead of full keypoints and Jacobians.The evaluation uses H.264, FOMM, and fs-vid2vid comparisons across varying encoded video qualities and sizes.
6. Conclusion
The paper presents a neural framework that decomposes unsupervised 3D keypoints into person-specific structure and motion transformations. This decomposition supports free-view synthesis and substantially more efficient live-video compression.
- Conclusion: Unsupervised 3D keypoints decompose the representation into person-specific canonical keypoints and motion-related transformations.The decomposition separates identity-related structure from motion information.
- Conclusion: Modifying only the keypoint transformation generates free-view videos, while transmitting only transformations improves compression ratios over existing methods.The conclusion connects both capabilities to live-video streaming and video conferencing.
- Conclusion: The framework is presented as a tool for streaming live videos with reduced bandwidth and a more immersive video-conferencing experience.This is the paper’s stated practical consequence.
A. Additional Network and Training Details
The model combines 3D appearance features and canonical keypoints from the source image with estimated pose and expression motion, then warps and renders them through neural components. Training uses reconstruction, perceptual, adversarial, equivariance, pose, deformation, and keypoint-prior losses.
- Network architectures: The appearance extractor maps source-image features into 3D features, while the canonical detector extracts unsupervised 3D keypoints through a U-Net-style architecture.The keypoint detector projects encoded features to 3D using a 1x1 convolution.
- Network architectures: The pose and deformation estimators predict rotation, translation, and expression deformations that provide motion-related geometry information.These estimators use ResNet bottleneck blocks followed by global pooling and separate linear layers.
- Motion field and generator: Source features are warped using flows from keypoint pairs, combined with learned masks, and passed to a 3D U-Net; an occlusion mask supports final image generation.The generator projects warped 3D features to 2D and applies residual blocks and upsampling layers.
- Losses: The loss suite includes multi-scale VGG perceptual loss, patch-GAN and feature-matching losses, and equivariance loss for consistent keypoint estimation.Equivariance training uses affine transformations and randomly sampled thin plate splines.
- Losses: Keypoint and deformation priors discourage nearby keypoints, depth drift, and excessively large expression deformations.The experiments set Dt to 0.1, zt to 0.33, and use an L1 deformation prior.
- Training: Training uses ADAM with learning rate 0.0002 on eight 32GB V100 GPUs, progressing from 256x256 images for 100 epochs to 512x512 fine-tuning for 10 epochs.Spectral normalization is applied to generator and discriminator layers, with synchronized BatchNorm in the generator.
B.1. Datasets
Evaluation uses VoxCeleb2 and TalkingHead-1KH, with the latter assembled as a large collection of high-quality talking-head videos from multiple sources. The datasets support training and validation across varied video quality and scale.
- VoxCeleb2: VoxCeleb2 contains about 1M celebrity talking-head videos, including 280K high-bit-rate training videos and a validation set of about 36K videos.The evaluation follows the original dataset’s training and test split.
- TalkingHead-1KH: TalkingHead-1KH combines about 1000 hours of high-resolution, high-bit-rate videos from YouTube, the Ryerson audio-visual dataset, and recordings made with subject permission.Its videos generally have higher resolution and better image quality than VoxCeleb2.
B.2. Metrics
The paper evaluates talking-head synthesis using fidelity, perceptual distribution, and landmark-based metrics, and reports comparative motion-transfer results.
- L1, PSNR, SSIM, and MS-SSIM quantify reconstruction faithfulness, while FID measures similarity between synthesized and real-video distributions.AKD measures the distance between facial landmarks detected in recreated and original videos.
- AKD evaluates semantic consistency by comparing facial landmarks extracted from recreated and original videos.
- Our method performs best in quantitative comparisons using relative motion for cross-identity motion transfer.
B.4. Ablation study
The ablations examine keypoint prediction, keypoint count, canonical representations, compression, adaptive bandwidth, and failure cases, showing benefits alongside occlusion limits.
- The preferred model works best across the alternatives in the ablation study.
- Two-step keypoint prediction is only slightly better quantitatively than direct prediction, but direct prediction lacks output head-pose control.
- Using more keypoints correlates positively with output quality.
- 53.03 B of per-frame metadata corresponds to 0.001618 bits per pixel after arithmetic compression with the adaptive scheme.
- Adaptive keypoint dropout reduces transmitted metadata when bandwidth varies, while large occlusions degrade synthesis quality.
C.4. Dataset
The test set contains 222 high-resolution talking-head videos collected from the web and manually checked for quality.
- 222 high-resolution talking-head videos form the test set, with heads at least 512x512 pixels.
- The videos contain a mean of 608 frames and a median of 661 frames, ranging from 20 to 1024 frames.
C.5. Additional experiment results
Additional compression experiments compare the adaptive method with codecs and synthesis baselines using perceptual quality and human preference evaluations.
- The adaptive method requires much lower bandwidth than FOMM, fs-vid2vid, H.264, and H.265 at the compared LPIPS rates.
- Human evaluations find visual quality comparable to H.264 at CRF 36 and H.265 at CRF 37, with a much lower bit-rate.