Source-linked AI summary
Mask-Guided Portrait Editing with Conditional GANs
Shuyang Gu, Jianmin Bao, Hao Yang, Dong Chen, Fang Wen, Lu Yuan
TL;DR
Existing portrait synthesis methods face challenges in diversity, fine-detail quality, and local controllability. This paper introduces a mask-guided conditional GAN with separate facial-component embeddings, achieving stronger mask-driven synthesis while supporting portrait editing and face-parsing augmentation.
Problem
Existing face-synthesis approaches struggle with diversity, fine details, hair and background quality, and controllable editing of local facial components.
Method
The framework uses mask-guided conditional GANs with local embeddings for individual facial components to generate controllable portrait images.
Results
Experiments show improved diversity and quality over state-of-the-art mask- or skeleton-driven face synthesis methods.
Takeaways & Limitations
The framework supports mask-to-face synthesis, local facial editing, face Swap+, and marginal face-parsing gains through data augmentation.
Abstract
from arXiv · showhide
Portrait editing is a popular subject in photo manipulation. The Generative Adversarial Network (GAN) advances the generating of realistic faces and allows more face editing. In this paper, we argue about three issues in existing techniques: diversity, quality, and controllability for portrait synthesis and editing. To address these issues, we propose a novel end-to-end learning framework that leverages conditional GANs guided by provided face masks for generating faces. The framework learns feature embeddings for every face component (e.g., mouth, hair, eye), separately, contributing to better correspondences for image translation, and local face editing. With the mask, our network is available to many applications, like face synthesis driven by mask, face Swap+ (including hair in swapping), and local manipulation. It can also boost the performance of face parsing a bit as an option of data augmentation.
1. Introduction
The paper targets diversity, quality, and controllability limitations in portrait synthesis and editing with a mask-guided conditional GAN framework. It separates facial-component embeddings and supports mask-driven synthesis, local editing, face swapping, and face-parsing augmentation.
- Problem: Existing portrait GANs struggle with fine facial details, hair, backgrounds, and limited diversity, motivating better control over local components.The paper identifies diversity, quality, and controllability as the central design requirements.
- Method: The proposed end-to-end framework uses local embeddings, mask-guided generation, and background fusion to synthesize and edit portraits.Five auto-encoders separately encode the left eye, right eye, mouth, skin and nose, and hair, while masks specify component regions.
- Results: The framework outperforms state-of-the-art mask- or skeleton-driven synthesis methods in diversity and quality.The authors also report marginally improved face parsing when synthesized facial data are used for augmentation.
- Applications: Mask conditioning enables mask-to-face synthesis, component-level edits, local appearance transfer, and face swapping that includes hair.Examples include removing hair, changing eye size, making a face smile, and modifying individual facial components.
2. Related Work
The related work spans GAN-based image translation, visual attribute transfer, face swapping, and non-parametric synthesis. The paper differentiates its approach through facial-instance embeddings that are recombined with masks for explicit face and hair manipulation.
- Mask-guided translation: Global mask- or skeleton-based translators offer realistic image-to-image translation but have limited diversity, especially in emotion, color, or illumination.The paper contrasts pix2pix, pix2pixHD, and BicycleGAN with its more component-specific representation.
- Attribute transfer and face swapping: The proposed method disentangles and recombines local facial-instance embeddings with masks, supporting explicit face and hair swapping.This distinguishes it from related visual attribute-transfer methods whose focus includes style, color, or makeup transfer.
- Non-parametric synthesis: Unlike non-parametric methods that warp and stitch image patches, this approach encodes facial instances as embeddings and generates new images through a mask-guided network.Joint training of the sub-networks is presented as a route to higher-quality facial images and transformations such as opening a previously closed mouth.
3. Mask-Guided Portrait Editing Framework
The framework combines component-level embeddings, mask-guided generation, background fusion, and multiple training losses for controllable portrait editing. Its architecture uses parsed facial regions and target masks to preserve source details, generate foreground faces, and reduce fusion artifacts.
- Framework Inputs: The framework uses source and target images with their masks, and manually editing the target mask allows changes to the desired facial structure.The four inputs are the source image and mask together with the target image and mask; masks can also come from a face parsing network.
- Local Embedding Sub-Network: Five auto-encoders separately embed the left eye, right eye, mouth, skin and nose, and hair, enabling component-level control and recombination.The source mask identifies each component before embedding; changing or recombining components is supported during generation.
- Mask-Guided Generative Sub-Network: The mask-guided generator places each learned component tensor at the corresponding target-mask location, concatenates them with mask features, and synthesizes the foreground face.Component tensors are copied into zero-filled spatial tensors centered on target component locations before concatenation.
- Background Fusing Sub-Network: The background-fusing sub-network encodes the target background and combines it with the generated foreground to reduce boundary artifacts from mismatched skin colors and imperfect hair masks.Directly copying the target background can create artifacts because it may contain neck skin and hair regions that do not match the generated face.
- Loss Functions: Training combines local reconstruction, global reconstruction, adversarial, feature-matching, and face-parsing losses to preserve components, reconstruct inputs, improve realism, and match target masks.The generator loss sums the component losses, while face parsing constrains generated samples to share the target mask and feature matching compares discriminator representations.
4. Experiments
Experiments evaluate the framework’s diversity, quality, mask preservation, and controllability across synthesis, editing, face swapping, and face parsing. The method produces realistic, diverse, mask-consistent faces, improves quality over variants, supports local edits and face swapping, and raises parsing accuracy with generated data.
- 4.1. Analysis of the Proposed Framework: The local embedding and background-fusing subnetworks improve generated-sample quality, with direct background copying otherwise causing noticeable boundary artifacts.The ablation evaluates 5,000 generated faces per setting using FID; the supplied passage reports the direction of improvement but not table values.
- 4.1. Analysis of the Proposed Framework: Face parsing loss preserves the target mask in generated samples, according to a comparison against a framework without that loss.
- 4.2. Mask-to-Face Synthesis: Generated faces conditioned on one target mask remain photo-realistic and diverse in skin color, hair color, eye makeup, and beards while maintaining the mask.
- 4.2. Mask-to-Face Synthesis: Our method produces realistic, clear, diverse mask-preserving faces, while BicycleGAN and pix2pixHD show diversity mainly in skin color or illumination.The comparison attributes the baselines’ limited diversity to global encoders that cannot leverage component-level facial variation.
- 4.3. Face Editing and Face Swap+: Local mask edits change hair, mouth, eyes, eyebrows, and emotion, while component replacement enables beards, altered mouth and hair colors, and eye makeup.
- 4.3. Face Editing and Face Swap+: Face swap+ transfers appearance while preserving component shape and explicitly supports hair swapping, producing reasonable results for large poses and extreme illumination.When glasses are labeled as background, the background-fusion subnetwork can reconstruct them.
- 4.4. Synthesized Faces for Face Parsing: Adding generated faces improves Helen face-parsing accuracy by 0.8% over using no generated face images.The evaluation uses 100 Helen test images.
5. Conclusion
The paper presents an end-to-end mask-guided conditional GAN framework for diverse, high-quality, and controllable facial image synthesis and portrait editing.
- The framework synthesizes diverse, high-quality, and controllable facial images from given masks while supporting high-level portrait editing.Users can edit facial components in generated faces and transfer local appearances from source images.