Source-linked AI summary
ReenactGAN: Learning to Reenact Faces via Boundary Transfer
Wayne Wu, Yunxuan Zhang, Cheng Li, Chen Qian, Chen Change Loy
TL;DR
Face reenactment must transfer expressions across people despite varied conditions, unseen inputs, and limited paired data. ReenactGAN uses boundary-space encoding, target-specific transformation, and target decoding to produce photo-realistic reenactments through a feed-forward pipeline. The approach supports arbitrary-source-to-target reenactment, while its boundary representation depends on suitable alignment or other annotated training data.
Problem
Face reenactment is difficult because direct GAN mappings must handle varied poses, expressions, lighting, unseen images, and a lack of pairwise data.
Method
ReenactGAN encodes faces into a boundary latent space, transforms source boundaries with a target-specific transformer, and decodes them into target faces.
Results
ReenactGAN performs photo-realistic face reenactment with accurate expressions and good quality, and its many-to-one transformer extends decoder capability.
Takeaways & Limitations
The feed-forward design supports reenactment from arbitrary source people to a specific target and is easier to re-implement and distribute.
Takeaways & Limitations
The boundary representation is not unique; other latent media require large datasets to constrain learning, while this method relies on available alignment data.
Abstract
from arXiv · showhide
We present a novel learning-based framework for face reenactment. The proposed method, known as ReenactGAN, is capable of transferring facial movements and expressions from monocular video input of an arbitrary person to a target person. Instead of performing a direct transfer in the pixel space, which could result in structural artifacts, we first map the source face onto a boundary latent space. A transformer is subsequently used to adapt the boundary of source face to the boundary of target face. Finally, a target-specific decoder is used to generate the reenacted target face. Thanks to the effective and reliable boundary-based transfer, our method can perform photo-realistic face reenactment. In addition, ReenactGAN is appealing in that the whole reenactment process is purely feed-forward, and thus the reenactment process can run in real-time (30 FPS on one GTX 1080 GPU). Dataset and model will be publicly available at https://wywu.github.io/projects/ReenactGAN/ReenactGAN.html
1 Introduction
The paper frames face reenactment as difficult because direct learning must cover varied poses, expressions, lighting, and unseen faces without paired data. ReenactGAN addresses this with a boundary latent space, target-specific transformation, and feed-forward decoding.
- Motivation: Direct GAN-based face reenactment struggles with extreme poses, large expressions, lighting variation, and unseen images.The paper identifies these conditions as difficult for limited-sample face-to-face mapping and reports CycleGAN failures in such cases.
- Training Advantage: Boundary-space learning can use paired face-and-boundary alignment data while keeping source and target video domains unpaired.The paper attributes this to the availability of large-scale face-alignment training sets.
- Framework: ReenactGAN maps source faces into boundary space, adapts that space to a target identity, and decodes the result into the target face.Its three components are an encoder, a target-specific transformer, and a target-specific decoder.
- Boundary Latent Space: Facial boundaries provide a compact, relatively identity-agnostic representation that preserves geometric information for expression transfer.The paper also states that boundary space is more robust to challenging poses, expressions, and lighting conditions.
- Implementation: All ReenactGAN components are feed-forward, making the framework easier to train and implement than traditional model-based methods.The paper presents this feed-forward design as a practical property of the proposed framework.
- Many-to-One Transfer: Target-specific transformers enable many-to-one reenactment from arbitrary source people, which conventional GAN image-to-image transfer cannot provide.The transformer adapts arbitrary source boundary spaces to a specific target.
2 Related Work
Prior face-reenactment studies are largely model-based and typically capture facial movement before fitting or transferring it to a face representation.
- Model-Based Approaches: Most existing face-reenactment studies use model-based approaches.These approaches include tracking face templates, matching optical-flow measurements, or capturing movements with RGB or RGB-D cameras.
- Movement Capture: Model-based pipelines first capture facial movements using templates, optical flow, or RGB/RGB-D cameras.Recent landmark detectors also support tracking facial components such as eyes and mouth.
- Landmark Tracking: Facial landmark detection enables tracking of specific input components, including the eyes and mouth.
3 Face Reenactment via Boundary Transfer
ReenactGAN separates face reenactment into boundary encoding, target-specific boundary transformation, and decoding. Its boundary latent space preserves facial structure while reducing sensitivity to identity, and transformer losses address cross-person shape mismatch.
- Framework: ReenactGAN uses a boundary encoder, target-specific many-to-one transformer, and target-specific decoder, with every component implemented as a feed-forward network.At test time, the query face passes sequentially through these components.
- Encoder and Decoder: The encoder maps faces from pixel space X into boundary space B, while each target-specific decoder maps B to that target person’s face subset.The encoder is shared, whereas decoders are trained for individual targets using target face examples.
- Boundary Latent Space: Boundary heatmaps represent contours of facial parts at 64 × 64 spatial resolution and are supervised with landmark-derived ground-truth heatmaps.The representation is designed to retain facial structure while being less affected by background clutter, lighting, and facial textures.
- Boundary Latent Space: Without boundary adaptation, a target-specific decoder can produce severe artifacts when source and target face boundaries differ substantially.Reconstruction quality remains comparable with and without the boundary latent space, while direct cross-person decoding fails under large structural gaps.
- Boundary Transformer: Target-specific transformers map arbitrary source boundaries into target boundary spaces, addressing the multi-valued inverse problem created by mapping the full boundary set to a target subset.Multiple transformers are trained jointly to transform the shared boundary space into each target-specific subset.
- Boundary Transformer: Transformer training combines cyclic consistency, adversarial discrimination of target poses, and a PCA-based shape constraint that preserves rough head motion.The shape constraint encourages transformed boundaries to follow their sources in a compressed PCA subspace.
4 Experiments
Experiments evaluate ReenactGAN’s image quality, facial-action consistency, and transformer design against CycleGAN and Face2Face, including unseen inputs. Results indicate stronger expression transfer, better handling of varied faces, and a crucial role for the transformer.
- Qualitative Comparisons with State-of-the-arts: ReenactGAN works well on profile faces and transfers expressions from unseen people more effectively than the compared methods.All three methods perform well on frontal faces; ReenactGAN additionally tracks global head pose, unlike Face2Face’s inner-face alterations.
- A User Study on Face Reenactment Quality: The user study compares seven settings using TrueSkill scores from 180 result groups, with 21 pairwise comparisons per group.The settings include ground truth, reconstruction, ReenactGAN with and without its transformer, and CycleGAN.
- Qualitative Comparisons with State-of-the-arts: Vanilla CycleGAN performs poorly on hair generation and fails on wild inputs because it learns a one-source-to-one-target transformation.
- A User Study on Face Reenactment Quality: ReenactGAN outperforms CycleGAN in the user study, while the transformer plays a crucial role and unseen wild data causes a performance drop.Despite the drop, ReenactGAN handles unseen wild inputs well.
- Facial Action Consistency: ReenactGAN transfers facial action responses more accurately than Face2Face, with significantly higher correlations to the source video.The comparison uses a facial action detector trained on 98 landmarks for 12 action units; its average DISFA cross-validation accuracy is 55.2%.
- Ablation Study on Boundary Transformer: Removing the transformer can produce blurred results or severe textural artifacts, while shape and cycle losses improve pose tracking and eyes and mouth generation.The shape constraint improves source-pose adherence over GAN loss alone, and the cycle loss further refines results.
5 Conclusion and Discussion
ReenactGAN performs photo-realistic face reenactment by bridging source and target identities through a boundary latent space and a many-to-one transformer. Experiments report accurate expressions and good quality, including for unseen people.
- ReenactGAN introduces a boundary latent space to bridge source and target domains for photo-realistic face reenactment.
- The many-to-one transformer extends the decoder’s capability, and the trained model can handle faces from unseen people.
- The authors identify facial component discriminators, decoder compression, and human–non-human reenactment as future improvements.