PaperlayerView exact revision on arXiv

Source-linked AI summary

MaskGAN: Towards Diverse and Interactive Facial Image Manipulation

Cheng-Han Lee, Ziwei Liu, Lingyun Wu, Ping Luo

arXiv:1907.11922v2cs.CVcs.GRcs.LG

TL;DR

Facial manipulation methods have limited attribute coverage or user interaction freedom. MaskGAN uses semantic masks with Dense Mapping Network and Editing Behavior Simulated Training, and it outperforms state-of-the-art methods on attribute transfer and style copy while introducing CelebAMask-HQ.

  • Problem

    Existing facial manipulation methods either operate on predefined face attributes or give users little freedom to interactively manipulate images.

  • Method

    MaskGAN combines Dense Mapping Network for mask-to-image style mapping with Editing Behavior Simulated Training for robustness to user mask changes.

  • Results

    MaskGAN shows superior performance over other state-of-the-art methods on attribute transfer and style copy.

  • Takeaways & Limitations

    Semantic masks provide MaskGAN with an intermediate representation for diverse, interactive face manipulation with fidelity preservation.

Abstract

from arXiv · show

Facial image manipulation has achieved great progress in recent years. However, previous methods either operate on a predefined set of face attributes or leave users little freedom to interactively manipulate images. To overcome these drawbacks, we propose a novel framework termed MaskGAN, enabling diverse and interactive face manipulation. Our key insight is that semantic masks serve as a suitable intermediate representation for flexible face manipulation with fidelity preservation. MaskGAN has two main components: 1) Dense Mapping Network (DMN) and 2) Editing Behavior Simulated Training (EBST). Specifically, DMN learns style mapping between a free-form user modified mask and a target image, enabling diverse generation results. EBST models the user editing behavior on the source mask, making the overall framework more robust to various manipulated inputs. Specifically, it introduces dual-editing consistency as the auxiliary supervision signal. To facilitate extensive studies, we construct a large-scale high-resolution face dataset with fine-grained mask annotations named CelebAMask-HQ. MaskGAN is comprehensively evaluated on two challenging tasks: attribute transfer and style copy, demonstrating superior performance over other state-of-the-art methods. The code, models, and dataset are available at https://github.com/switchablenorms/CelebAMask-HQ.

1. Introduction

MaskGAN addresses limited user freedom and predefined attributes in facial manipulation by using semantic masks for diverse, interactive editing. Its DMN, EBST strategy, and CelebAMask-HQ dataset support this approach.

  • Existing facial manipulation methods either use predefined attributes or provide limited freedom for interactive image editing.
  • MaskGAN uses semantic masks as an intermediate representation for flexible manipulation while preserving image fidelity.Users can specify facial component categories, shapes, and locations through modified masks.
  • Dense Mapping Network learns style mapping between target images and user-modified semantic masks, while EBST models user editing behavior.DMN includes an image generation backbone and spatial-aware style encoder.
  • EBST introduces dual-editing consistency and improves robustness to varied user-input mask changes during inference.Its training uses a pretrained DMN, MaskVAE, and alpha blender sub-network.
  • MaskGAN outperforms other state-of-the-art methods on attribute transfer and style copy tasks.The evaluation covers two challenging facial manipulation tasks.
  • CelebAMask-HQ provides over 30,000 512×512 face images with fine-grained masks covering 19 facial component categories.The dataset was constructed to facilitate large-scale studies.
  • The paper presents MaskGAN, EBST, and CelebAMask-HQ as contributions intended to support interactive editing, robust mask manipulation, and geometry-oriented research.

2. Related Work

Prior facial manipulation work includes semantic-level editing and geometry-level manipulation. Existing methods include conditional translation, attribute disentanglement, exemplar-based generation, and latent-code exchange.

  • Semantic-level face manipulation methods have produced impressive results using approaches such as inverse mappings, attribute disentanglement, and multi-domain translation.
  • Some geometry-level methods transfer facial attributes between exemplars, including approaches that exchange latent codes between two faces.The cited geometry-level methods address instance-level attribute transfer.

3. Our Approach

MaskGAN performs structural face manipulation by mapping a target image and semantic masks to synthesized faces, while EBST improves robustness to user-modified masks. Its multi-objective training combines adversarial, feature-matching, and perceptual losses for high-resolution synthesis.

  • Overall Framework: MaskGAN uses a target image, target semantic mask, and user-modified source mask to synthesize a manipulated output face.The framework includes DMN, MaskVAE, and Alpha Blender, trained with EBST.
  • Dense Mapping Network: Dense Mapping Network combines a Spatial-Aware Style Encoder with an Image Generation Backbone to map target-image style onto corresponding regions of the source mask.The encoder uses both target-image appearance and target-mask spatial information before synthesis.
  • Editing Behavior Simulated Training: MaskVAE models structural-prior manifolds, enabling smooth mask interpolation that supports EBST's locally perturbed mask inputs.EBST uses interpolated masks and an Alpha Blender to retain subject appearance and identity information.
  • Editing Behavior Simulated Training: EBST trains DMN with dual edited masks and blending, making the generator more robust to changes in user-modified masks during inference.The procedure alternates updates to the DMN and the combined model until convergence.
  • Evaluation: On facial attribute transfer and style copy, MaskGAN is reported to outperform other state-of-the-art methods.The supplied figure captions specifically discuss attribute preservation differences in smiling transfer and style copy.
  • Multi-Objective Learning: The training objective combines conditional adversarial, feature-matching, and perceptual losses, while multi-scale discriminators support high-resolution image synthesis.The feature and perceptual weights are both set to 10 through cross validation.

4. CelebAMask-HQ Dataset

CelebAMask-HQ is a large-scale, high-resolution facial semantic-label dataset built from CelebA-HQ. It provides detailed annotations, quality control, and explicit handling of occluded facial components.

  • Dataset Construction: CelebAMask-HQ contains 30,000 high-resolution face images labeled with semantic masks.The dataset is based on CelebA-HQ images from CelebA.
  • Annotations: Each 512 × 512 image has hand-annotated labels covering 19 facial components and accessories.Classes include skin, nose, eyes, eyebrows, ears, mouth, hair, hat, eyeglasses, and clothing.
  • Label Size Selection: The dataset uses 512 × 512 labels because annotating at 1024 × 1024 would be costly, while nearest-neighbor interpolation can extend labels without noticeable artifacts.The original CelebA-HQ images were 1024 × 1024.
  • Quality Control: Every segmentation mask undergoes quality-control checks, followed by multiple refinement rounds by annotators.The process is applied after manual labeling.
  • Amodal Handling: Partly occluded components are labeled by inferring hidden regions, while totally occluded components are left unannotated.This is the dataset's stated occlusion-handling policy.
  • Dataset Comparison: CelebAMask-HQ is compared with the Helen dataset through dataset statistics.The comparison is reported in Table 5.

5. Experiments

MaskGAN is evaluated across attribute transfer, style copy, identity preservation, and interactive editing using quantitative, visual, and human-perception measures. The experiments compare MaskGAN with several state-of-the-art methods and examine the effects of its architectural components and editing-behavior training.

  • Attribute Transfer: MaskGAN achieves relatively high classification and segmentation accuracy for smiling transfer, although it does not obtain the best FID score.StarGAN and ELEGANT have better FID, while Pix2PixHD-m has the best classification accuracy but inferior FID.
  • Attribute Transfer: MaskGAN provides plausible smiling-transfer results, whereas StarGAN fails in the smiling region, ELEGANT transfers smiling inconsistently, and Pix2PixHD-m has lower perceptual quality.SPADE has poor attribute-keeping ability in the visual comparison.
  • Style Copy: For style copy, MaskGAN transfers makeup and gender styles better visually than SPADE and Pix2PixHD-m, while SPADE obtains better segmentation accuracy.MaskGAN also has relatively high attribute classification accuracy compared with Pix2PixHD-m.
  • Ablation Study: Editing Behavior Simulated Training improves robustness to structure changes in user-modified masks and strengthens attribute-keeping evaluation scores.The ablation attributes this improvement to reducing content-generation influence from structural changes during inference.
  • Interactive Face Editing: MaskGAN supports interactive geometry-level editing of facial-component shape, location, and category, including accessories, face and nose shape, and hair.The interactive results include a hair-color failure case when adding hair to a short-hair woman.

6. Conclusions

The paper presents MaskGAN as a geometry-oriented framework for flexible face manipulation and contributes a high-resolution, fine-grained mask dataset. It reports evaluation on attribute transfer and style copy and identifies image completion as future work for preserving unedited details.

  • Conclusions: MaskGAN uses Dense Mapping Network and Editing Behavior Simulated Training to support geometry-oriented face manipulation through semantic masks.The framework targets flexible manipulation with fidelity preservation.
  • Conclusions: The framework is evaluated on attribute transfer and style copy, where it shows superior performance over other state-of-the-art methods.The conclusion summarizes both challenging evaluation tasks at the paper level.
  • Conclusions: CelebAMask-HQ contributes a large-scale, high-resolution face dataset with fine-grained mask annotations.The dataset is presented as an additional contribution alongside the manipulation framework.
  • Conclusions: Future work includes combining MaskGAN with image completion techniques to preserve details in regions without editing.This is the paper’s stated future direction.

A. Additional Implementation Details

The implementation uses a modular architecture built around MaskVAE, Dense Mapping Network, Alpha Blender, and a discriminator. Dense Mapping Network itself combines an image-generation backbone with a spatial-aware style encoder.

  • Network Architecture: MaskGAN comprises MaskVAE, Dense Mapping Network, Alpha Blender, and Discriminator.Dense Mapping Network contains the Image Generation Backbone and Spatial-Aware Style Encoder.

B. Additional Ablation Study

Additional ablations examine spatial feature transformation and Editing Behavior Simulated Training. The reported comparisons associate SFT layers with stronger fusion effects and EBST with improved style-copy appearance and attribute keeping.

  • Spatial Feature Transform: Spatial Feature Transform layers use more parameters to fuse different domains and are reported to work better than concatenation.The comparison is summarized quantitatively in Table 5.
  • Editing Behavior Simulated Training: Editing Behavior Simulated Training produces better color saturation and attribute-keeping quality for style copy.The visual comparison specifically mentions heavy makeup as an attribute example.

C. Additional Visual Results

This section presents additional visual comparisons for smiling attribute transfer, style copy, EBST, and style-copy ablations. It also demonstrates an interactive interface in which users edit facial components or add accessories.

  • Attribute Transfer: Figures 14–17 show additional visual results for smiling attribute transfer against Pix2PixHD, ELEGANT, and StarGAN.The figures include models trained at 256 × 256 resolution, marked with an asterisk.
  • EBST: Figure 13 provides visual comparisons between training with and without Editing Behavior Simulated Training.
  • Ablation Study: Table 5 reports a style-copy ablation using attribute-classification accuracy for Male, Heavy Makeup, and No Beard.Its train/test split differs from the main paper.
  • Style Copy: Figures 18–21 show additional visual results for style copy compared with modified Pix2PixHD.
  • Interactive Interface: The accompanying interface lets users edit facial-component shapes or add accessories by manipulating the semantic segmentation mask.
Loading 1907.11922v2…