Source-linked AI summary
SC-FEGAN: Face Editing Generative Adversarial Network with User's Sketch and Color
Youngjoo Jo, Jongyoul Park
TL;DR
Image completion systems often lacked direct control over the shape and color of large erased regions, while existing approaches could produce awkward edges or struggle with extensive edits. SC-FEGAN uses an end-to-end convolutional network with free-form masks, sketches, and colors, plus SN-patchGAN and style loss. The authors report high-quality, realistic edits of large regions in one pass with minimal user effort.
Problem
Existing image completion methods can be limited to low-resolution images, produce awkward masked-region edges, and lack user input to guide large-region synthesis.
Method
SC-FEGAN uses an end-to-end fully convolutional U-Net-like network with gated convolutions, free-form masks, sketches, color data, SN-patchGAN, and additional style loss.
Results
The authors report high-quality and realistic editing of large regions in one pass, including detailed synthetic hairstyles and earrings, with improved structure and shape over Deepfillv1.
Takeaways & Limitations
SC-FEGAN supports intuitive face-image modification by allowing users to specify both the structure and color of erased regions.
Takeaways & Limitations
Earlier completion systems were limited to low-resolution images and could produce awkward edges in masked regions.
Abstract
from arXiv · showhide
We present a novel image editing system that generates images as the user provides free-form mask, sketch and color as an input. Our system consist of a end-to-end trainable convolutional network. Contrary to the existing methods, our system wholly utilizes free-form user input with color and shape. This allows the system to respond to the user's sketch and color input, using it as a guideline to generate an image. In our particular work, we trained network with additional style loss which made it possible to generate realistic results, despite large portions of the image being removed. Our proposed network architecture SC-FEGAN is well suited to generate high quality synthetic image using intuitive user inputs.
1. Introduction
SC-FEGAN addresses limitations of conventional image completion by using free-form masks, sketches, and colors as user guidance. Its architecture and losses target realistic, high-resolution editing of large erased regions.
- Demonstrated editing: The system is demonstrated on edits that can change facial shape, color, hairstyle, eyes, and earrings using free-form user input.Figure 1 presents these editing cases as examples of the system’s interactive capabilities.
- Motivation: Conventional completion methods restore square-masked regions but can be limited to low-resolution images, produce awkward edges, and lack user guidance.These limitations reduce control over the shape and color of synthesized regions.
- Proposed system: SC-FEGAN is an end-to-end trainable fully convolutional network that accepts free-form masks, sketches, and color inputs.The system is designed to incorporate both the user’s intended structure and color.
- Architecture: A U-Net-like architecture with gated convolutions is reported to produce superior and more detailed results than a Coarse-Refined network in the authors’ case.The authors also describe this architecture as easier and faster for training and inference.
- Training objectives: SN-patchGAN and an additional style loss target awkward mask edges and realistic editing when large portions of the image are erased.The authors associate these components with detailed outputs such as hairstyles and earrings.
2. Related Work
Related work established deep-learning methods for image translation and completion, but prior systems differed in their training requirements and user controls. SC-FEGAN extends this line by combining free-form masks with both sketch and color inputs.
- Interactive editing: Traditional image modification commonly relies on handcrafted operations that require expert knowledge and substantial time for delicate results.This motivates methods that can incorporate user intent more directly.
- Image translation: Pix2Pix learns paired domain transformations, whereas CycleGAN removes the requirement for paired target images.These methods frame image translation as learning mappings between image domains.
- User-guided translation: User-guided translation methods feed inputs such as domain labels or desired colors to generators to influence synthesized images.StarGAN and Ideepcolor illustrate flexible domain translation and color-guided monochrome conversion.
- Image completion: Image completion must both fill deleted regions and properly reflect the user’s input in those restored areas.Earlier GAN completion systems used U-Net generators with local and global discriminators to assess reconstructed content.
- SC-FEGAN’s position: Deepfillv2 introduced free-form masks, SN-patchGAN, and gated convolutions, enabling the network to reflect sketch input in generated results.SC-FEGAN builds on this direction by adding color data while retaining a U-Net structure.
3. Approach
SC-FEGAN is an end-to-end face-editing system trained on free-form masks, sketches, and colors, using a U-net generator and adversarial, perceptual, style, and reconstruction losses.
- Training Data: SC-FEGAN prepares 512×512 CelebA-HQ training data with free-form masks, including eye-position and randomly added hair masks.The dataset uses 29,000 training images and 1,000 test images; sketches are generated with HED and color domains with GFC segmentation and median colors.
- Training Data: The training inputs combine an incomplete RGB image, binary sketch, RGB color stroke map, binary mask, and noise.The generator encoder receives these inputs as a 512×512×9 tensor.
- Network Architecture: The generator uses a U-net with gated 3×3 convolutions, seven downsampling stages, dilated convolutions, transposed-convolution upsampling, and skip connections.It contains 16 convolution layers and outputs a same-size 512×512 RGB image.
- Network Architecture: The discriminator uses an SN-PatchGAN structure with 3×3 convolutions and gradient penalty loss, while the generator output uses tanh activation.LRN is applied after convolution layers except the input and output layers.
- Loss Functions: Generator training combines GAN, pixel-wise, perceptual, style, and total-variation losses to improve reconstruction, realism, and artifact suppression.Perceptual loss uses VGG-16 feature maps, style loss uses Gram matrices, and total variation loss targets checkerboard artifacts.
- Comparison: Compared with the Coarse-Refined network, the U-net produced less blurred results in the reported eye-region removal experiment.The authors attribute the system’s stronger performance in this regard to perceptual and style losses.
4. Results
The experiments compare architectures, losses, and related systems, then demonstrate SC-FEGAN’s face-editing and restoration capabilities. Results emphasize realistic structure and shape, large-region editing, and synthesis from sparse user guidance.
- Experimental setup: 44ms on GPU and 53ms on CPU are required for average 512×512 inference, regardless of input size and shape.Experiments used NVIDIA Tesla V100 hardware and a Power9 CPU.
- Ablation studies: The Coarse-Refined network’s refining stage blurred outputs because its L1 loss was smaller than the coarse network’s loss.The authors also note that this structure requires substantial memory and training time.
- Ablation studies: Without VGG loss, the network encountered problems similar to FaceShop, while perceptual and style losses improved large erased-region editing.The comparison is shown in Figure 5, including cases involving huge erased regions such as whole hair regions.
- Comparisons: Compared with Deepfillv1, SC-FEGAN produced better qualitative results for structure and shape with free-form masks.The comparison was conducted on the CelebA-HQ validation sets.
- Face image editing and restoration: SC-FEGAN changed face shape, color, eyes, and hair, including hairstyle synthesis when the entire hair region was erased.The system tolerated small drawing errors and generated realistic composites reflecting shadows and shapes in detail.
- Interesting results: Using HED-derived training masks over all areas, the network learned small details and generated reasonable facial images with limited input, including earrings.The authors describe these as results influenced by the training dataset and illustrate them in Figure 8.
5. Conclusions
SC-FEGAN is presented as an end-to-end generative image-editing system for free-form masks, sketches, and colors. The authors report high-quality, realistic editing across high-resolution face imagery, including large-region restoration with minimal user effort.
- SC-FEGAN provides free-form mask, sketch, and color inputs through an end-to-end trainable generative network.
- The system was trained on high-resolution CelebA-HQ imagery and produced successful, realistic editing results in many cases.
- The network modifies and restores large image regions in one pass while requiring minimal user effort.