Source-linked AI summary
X2Face: A network for controlling face generation by using images, audio, and pose codes
Olivia Wiles, A. Sophia Koepke, Andrew Zisserman
TL;DR
Controlling a face’s pose and expression from another face or modality is challenging without an explicit face representation. X2Face learns this representation self-supervised from video, enabling robust face driving and generation conditioned on audio or pose information without further training.
Problem
The paper asks whether face pose and expression can be controlled without explicitly modelling the face, using a representation learned self-supervised from video.
Method
X2Face learns face variation from video with self-supervised training, then drives a source identity using another face, audio, or pose codes.
Results
X2Face supports unconstrained face driving and can condition generation on audio or head pose without annotations or further training.
Takeaways & Limitations
The framework provides a versatile video face-editing tool for driving faces across identities and modalities in unconstrained settings.
Takeaways & Limitations
Generation quality is lower than approaches specifically designed for face transformation, and audio-driven generation can exhibit jitter.
Abstract
from arXiv · showhide
The objective of this paper is a neural network model that controls the pose and expression of a given face, using another face or modality (e.g. audio). This model can then be used for lightweight, sophisticated video and image editing. We make the following three contributions. First, we introduce a network, X2Face, that can control a source face (specified by one or more frames) using another face in a driving frame to produce a generated frame with the identity of the source frame but the pose and expression of the face in the driving frame. Second, we propose a method for training the network fully self-supervised using a large collection of video data. Third, we show that the generation process can be driven by other modalities, such as audio or pose codes, without any further training of the network. The generation results for driving a face with another face are compared to state-of-the-art self-supervised/supervised methods. We show that our approach is more robust than other methods, as it makes fewer assumptions about the input data. We also show examples of using our framework for video face editing.
1 Introduction
X2Face investigates lightweight, controllable animation of a still face by separating source identity from driving pose and expression. The model can use face, pose, or audio inputs without expression or pose labels during training.
- Motivation: Animating a still face controllably is useful for image editing, enhancement, and interactive systems, but requires representing the face and mapping control onto that representation.The paper frames controllable face animation as challenging because it may require approaches such as 3D modelling.
- Training: X2Face is trained without expression or pose labels.This training characteristic is stated in the model overview.
- X2Face framework: X2Face controls a source face instantiated from one or more frames using a driving frame, pose information, or audio information.Source frames come from the same face track, while the driving frame may come from the same or another video face track.
- X2Face framework: The generated frame preserves the source face’s identity and hairstyle while adopting properties specified by the driving vector.The supplied passage gives pose as an example of a property transferred from the driving input.
2 Related work
Related work spans explicit 3D face modeling, landmark-based 2D warping, learning-based image generation, and multimodal control. The proposed method combines self-supervision with control from other modalities without explicit face modeling.
- Explicit modelling of faces for image generation: Traditional facial animation fits a 3D morphable model (3DMM) and modifies its estimated parameters, with later work adding details, images, scans, or RGB-based parameter learning.A survey of this literature is referenced in the passage.
- Explicit modelling of faces for image generation: 3DMM or mesh methods transfer source expressions toward a driving face but require extra steps for hidden regions such as teeth.A neural network conditioned on one driving image can predict higher-level details for hidden-region completion.
- Explicit modelling of faces for image generation: A 2D warping approach uses facial landmarks to transfer one face’s expression onto another from a single source image while monitoring driving-video changes for fine-scale details.The method was motivated by limitations of morphable models in representing the full range of expressions, deformations, and higher-level details.
- Learning based approaches for image generation: Supervised image-generation methods condition outputs on known pose, expression, or landmark information, requiring training data with corresponding labels.The passage contrasts these methods with self-supervised approaches but ends before describing the full labeling requirement.
- Using multi-modal setups to control image generation: Audio-driven systems have controlled mouths, heads, or body movement, while the proposed method enables self-supervised multimodal control without explicitly modeling the face.The stated benefit is applicability to other domains.
3 Method
X2Face uses separate embedding and driving networks to transform a source frame into an embedded face and generate a frame controlled by the driving frame. Training proceeds through a two-stage curriculum combining self-supervised pixel reconstruction with identity-based finetuning.
- 3.1 Network architecture: X2Face takes a source frame and a driving frame as inputs to separate embedding and driving networks.The embedding network maps the source to an embedded face, while the driving network uses that representation to produce the generated frame.
- 3.1 Network architecture: The embedding network learns a bilinear sampler that maps pixels from the source frame to an embedded face representation.Its architecture is based on U-Net and pix2pix.
- 3.1 Network architecture: The driving network samples from the embedded face to generate an output with the driving frame’s pose and expression.Because it does not know the source frame’s pose or expression, the embedded face must provide a common representation across source frames.
- 3.2 Curriculum strategy: Training uses two stages: fully self-supervised training followed by finetuning with additional constraints from a CNN pretrained for face identification.Stage I uses only a pixelwise L1 loss between generated and driving frames.
- 3.2 Curriculum strategy: Identity losses compare generated frames with source or driving identities using feature-space similarity from a pretrained identity network.For a random-identity driving frame, the generated frame should retain the source identity while adopting the driving pose and expression.
4 Controlling the image generation with other modalities
A trained X2Face network can control a source face using modalities such as pose codes or audio instead of a driving frame. Pose control uses learned mappings between head pose and driving vectors, while audio control maps audio features to driving vectors.
- Overview: A trained X2Face network can use audio or pose as alternative modalities for controlling the source face.The driving vector is modified according to the non-visual input before decoding.
- Pose control: Varying pose-code pitch, yaw, or roll angles makes the generated source face vary accordingly.A forward mapping fp→v converts head pose p into driving vector v for the decoder.
- Pose control: Because VoxCeleb lacks ground-truth head pose, fv→p estimates source pose from the source driving vector.The mapping regresses pose from driving vectors using an annotated image-to-pose dataset, a fully connected layer, and L1 loss.
- Pose control: The pose-to-vector mapping fp→v regresses driving vectors from poses using a biased fully connected linear layer followed by batch normalization.After fv→p is learned, training pairs can be generated from VoxCeleb images and their inferred poses.
- Audio control: Audio features can drive a source face by mapping features from another frame to a corresponding driving vector.The forward mapping fa→v is trained on pairs of audio features and driving vectors from VoxCeleb videos.
5 Experiments
Section 5 evaluates X2Face through training and testing ablations, face-driven generation, pose-code control, and audio-driven generation. The experiments show that curriculum training, multiple source views, pose information, and audio can support realistic or reasonably aligned face generation without pose or expression annotations.
- Experimental setup: X2Face is evaluated using VoxCeleb videos, 256 × 256 dlib face crops, and randomly split train/validation/test identities.The reported split is 75/15/10.
- Ablation study: Both curriculum training and additional source views at test time improve reconstructed images on a 120K-pair VoxCeleb test evaluation.The driving frame comes from the same video as the source frames, so the target is the driving frame.
- Face-driven generation: X2Face transfers a driving video’s pose and expression onto source faces without expression or pose annotations while preserving realistic generation.The evaluation compares X2Face with CycleGAN and a supervised method designed top down.
- Face-driven generation: Compared with Averbuch-Elor et al., X2Face handles more significant pose changes and requires fewer assumptions about initial poses, expressions, and driving frames.X2Face can also be used with a single driving frame.
- Pose control: The 128D driving vector encodes head pose, with a linear predictor performing only slightly worse than a supervised method trained for pose regression.Head pose is represented by yaw, pitch, and roll, and the learned mappings generate frames while varying individual angles.
- Audio control: Despite an LRW-to-VoxCeleb audio domain gap and background noise, X2Face produces mouth movements reasonably close to those expected from corresponding sounds.Audio features are extracted from a 0.2s signal centered on the driving frame.
6 Using the embedded face for video editing
The embedded face is used for video editing by treating it as a pose- and expression-invariant UV texture map that can be drawn on directly.
- Embedded face as a texture map: The embedded face is expected to remain invariant to pose and expression, supporting its interpretation as a UV texture map of the face.This interpretation is inspired by the concept of an unwrapped mosaic.
- Editing procedure: For video editing, a source frame or set of source frames is extracted and input to the embedding network.
7 Conclusion
X2Face is a self-supervised, robust, and versatile framework for driving face generation across identities and modalities, and it can support video editing without annotated pose, landmarks, or depth data. Its generation quality remains below methods specifically designed for face transformation, motivating future work to retain its advantages while improving quality.
- Contributions: X2Face drives face generation using another face without assumptions about input pose, expression, or identity, making it more robust in unconstrained settings.The framework supports unseen identities.
- Contributions: After training, X2Face can drive faces using audio or head pose information with minimal alteration and can function as a video editing tool.The model is conditioned on modalities beyond another face.
- Contributions: X2Face achieves this functionality through self-supervised training on a large collection of videos without annotations for head pose, facial landmarks, or depth data.The training avoids requiring those annotations.
- Limitations and future work: Its generation quality is lower than approaches specifically designed for transforming faces, despite the method’s robustness, versatility, and multimodal conditioning.The limitation motivates improving generation quality while retaining these properties.
A Additional details on architectures and training.
The appendix provides additional architectural details for X2Face and reports qualitative results involving test-time views and the curriculum strategy.
- Architectures: Sections A.1 and A.2 detail X2Face’s embedding and driving subnetworks.These sections provide additional architectural information about the two subnetworks.
- Qualitative results: Section A.3 presents qualitative results using additional views at test time.The appendix examines the effect of incorporating additional views during testing.
- Qualitative results: Section A.3 also reports qualitative results for the curriculum strategy.The curriculum strategy is discussed alongside the test-time additional-view results.
A.1 Embedding network · A.2 Driving network
The embedding network adapts pix2pix’s U-Net to produce a 2-channel sampling map for embedding a source face. The driving network uses a skip-free encoder-decoder whose encoder produces a 128D driving vector.
- A.1 Embedding network: The embedding network is based on pix2pix’s U-Net architecture, with its final layer changed to two channels.Its output has dimensions 2 × 256 × 256.
- A.1 Embedding network: The resulting 2 × 256 × 256 output provides the sampler used to sample the source frame and obtain the embedded face.The exact architecture is given in Fig. 9.
- A.1 Embedding network: The embedding network encoder applies leaky ReLU activations with factor 0.2 and batch normalization after each convolution, except the first.Convolutional filters are 4 × 4 with stride/padding 2/1.
- A.1 Embedding network: Its decoder uses skip connections from encoder layers, doubling channel size through concatenation.Following each decoder convolution are ReLU, bilinear upsampling, batch normalization, and concatenation.
- A.2 Driving network: The driving network is an encoder-decoder with the same layer sizes as the embedding network but without skip connections.Its architecture is shown in Fig. 10.
- A.2 Driving network: 128D is the dimensionality of the driving vector produced by the driving network’s encoder.The driving vector corresponds to the encoder portion’s output.
- A.2 Driving network: The driving network encoder uses leaky ReLU with factor 0.2 and batch normalization after each convolution, except the first.It uses 4 × 4 convolutional filters with stride/padding 2/1, and its decoder has no skip connections.
- A.2 Driving network: The driving network decoder applies ReLU, bilinear upsampling, and batch normalization after each convolutional layer.The figure specifies a channel size of 128 for the driving vector.
A.3 Qualitative results for different training/testing setups
The paper evaluates four training/testing setups that vary training losses and the number of source frames. Multiple source frames improve robustness, quality, and sharpness, while photometric L1-only training still produces visually convincing results.
- Training/testing setups: The evaluation compares four setups combining stage I or II training with single-source or multi-source testing.Stage I uses photometric L1 loss only; stage II adds identity losses, and testing uses either one or multiple source frames.
- Multi-source testing: Testing with multiple source frames makes generated frames more robust to the choice of source frame.The generated frame is less unstable when the source frame changes.
- Multi-source testing: Multiple source frames improve the quality and sharpness of generated frames.This improvement accompanies the increased robustness observed in Fig. 11.
- Training losses: Photometric L1-only training produces visually similar and convincing results, although adding identity losses improves quantitative results.The visual results remain strong even without the identity loss functions described in the paper.
B Additional qualitative results
Additional qualitative results compare X2Face with CycleGAN, demonstrate head-pose control using a pose vector, and show video-editing applications.
- Training and testing settings: The paper compares training and testing settings using single or multiple source frames and shows generated frames across training stages and testing strategies.Fig. 11 includes the corresponding embedded faces and uses a driving frame from the same video as ground truth.
- Pose control: Additional results demonstrate controlling a source face’s head pose with a pose vector.The three head-pose angles are controlled in Fig. 15.
- Video editing: Further qualitative results apply X2Face to video editing by modifying an embedded face and controlling it with a sequence of driving frames.The application is illustrated using the Harry Potter scar in Figs. 13 and 14.
- Comparison with CycleGAN: X2Face preserves the source hair and background when compared with CycleGAN, producing temporal coherence across generated frames.The comparison is shown in Fig. 12.