Source-linked AI summary
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs
Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, Bryan Catanzaro
TL;DR
Prior methods struggled to generate high-resolution conditional-GAN images with fine details and realistic textures. This paper introduces a robust adversarial framework with multi-scale architectures and instance-aware, diverse synthesis, producing 2048 × 1024 photo-realistic images that outperform existing approaches in quantitative and human studies.
Problem
Conditional GAN methods struggled to generate high-resolution images with fine details and realistic textures.
Method
The paper combines a robust adversarial objective, multi-scale generator and discriminator architectures, instance-level segmentation, and feature embeddings for diverse interactive synthesis.
Results
2048 × 1024 photo-realistic images outperform existing methods in quantitative evaluations and human perception studies.
Takeaways & Limitations
The framework supports interactive semantic editing, including adding or removing objects, changing object types, and varying object appearance.
Abstract
from arXiv · showhide
We present a new method for synthesizing high-resolution photo-realistic images from semantic label maps using conditional generative adversarial networks (conditional GANs). Conditional GANs have enabled a variety of applications, but the results are often limited to low-resolution and still far from realistic. In this work, we generate 2048x1024 visually appealing results with a novel adversarial loss, as well as new multi-scale generator and discriminator architectures. Furthermore, we extend our framework to interactive visual manipulation with two additional features. First, we incorporate object instance segmentation information, which enables object manipulations such as removing/adding objects and changing the object category. Second, we propose a method to generate diverse results given the same input, allowing users to edit the object appearance interactively. Human opinion studies demonstrate that our method significantly outperforms existing methods, advancing both the quality and the resolution of deep image synthesis and editing.
1. Introduction
The paper introduces a learned approach for producing high-resolution images from semantic label maps, addressing GAN resolution challenges and realism limitations. It also supports interactive object manipulation and diverse appearance editing through instance-level segmentation and varied outputs.
- Motivation: Photo-realistic rendering traditionally requires explicit simulation of geometry, materials, and light transport, making virtual-environment construction expensive and time-consuming.A learned rendering model could reduce the need to model every aspect of the world explicitly.
- Motivation: The method generates high-resolution images from semantic label maps, which can support synthetic recognition data and editable scenario creation.Semantic labels are easier to create for desired scenarios than training images, and images can be transformed into the label domain for editing.
- Problem and approach: Prior approaches use conditional GANs or modified perceptual losses, but high-resolution synthesis can be unstable or produce images lacking fine details and realistic textures.The paper targets both GAN difficulty at high resolution and realism limitations in previous high-resolution results.
- Core method: 2048 × 1024 photo-realistic images are synthesized using a robust adversarial objective with multi-scale generator and discriminator architectures.The paper presents these components as addressing high-resolution generation and realism challenges.
- Interactive semantic manipulation: Instance-level segmentation enables adding, removing, and changing object types, while diverse generation allows interactive editing of an object’s appearance from the same label map.The framework is extended in two directions to support interactive semantic manipulation.
- Evaluation: The method outperforms state-of-the-art visual synthesis systems in quantitative evaluations and human perception studies, with additional ablations and edge2photo tests demonstrating broader analysis.The studies examine training objectives, instance-level segmentation, and generalizability beyond semantic manipulation.
2. Related Work
Prior work established GANs and adversarial learning for image synthesis, translation, and manipulation, but high-resolution conditional generation remained difficult and existing visual-processing methods often lacked interactive interfaces.
- Generative adversarial networks: GANs model natural image distributions and support applications including image generation, representation learning, manipulation, object detection, and video.Coarse-to-fine schemes have been proposed to synthesize larger images, such as 256 × images.
- Image-to-image translation: Adversarial learning is widely used for image-to-image translation because it avoids the blurry results often associated with L1 loss.The discriminator learns a trainable loss that adapts to differences between generated and real target-domain images.
- Image-to-image translation: Conditional GANs were considered difficult to train at high resolution because of training instability and optimization issues.Chen and Koltun instead used perceptual-loss regression to synthesize 2048 × 1024 images, which often lacked fine details and realistic textures.
- Deep visual manipulation: Deep neural networks have achieved promising results in style transfer, inpainting, colorization, and restoration, but most methods lack interfaces for adjusting results or exploring outputs.GAN-based optimization was developed to edit object appearance using learned priors.
3. Instance-Level Image Synthesis
This section presents a conditional adversarial framework for generating high-resolution photo-realistic images from semantic label maps. It improves the baseline pix2pix framework through enhanced objective and network design, instance-level semantic information, and feature embeddings for interactive object editing.
- Framework: The framework generates high-resolution photo-realistic images from semantic label maps using conditional adversarial training.It builds on a conditional GAN framework for image-to-image translation.
- Baseline and Improvements: Improved objective functions and network design increase the photorealism and resolution of generated results over the pix2pix baseline.pix2pix uses a generator to translate semantic label maps into realistic-looking images and a discriminator to distinguish real from translated images.
- Instance-Level Controls: Instance-level object semantics improve image quality, while feature embeddings address multimodal synthesis and enable interactive object editing.The embedding scheme is designed to better handle multiple possible image outcomes from the same semantic input.
D LGAN(G, D) (1)
The method extends pix2pix for high-resolution synthesis with a coarse-to-fine generator, multi-scale discriminators, and a feature-matching adversarial loss. These components address unstable training, limited receptive fields, and insufficient realism at high resolution.
- Motivation: Directly applying pix2pix to high-resolution Cityscapes images caused unstable training and unsatisfactory image quality, with generated resolution limited to 256 × 256.The original framework uses a U-Net generator and patch-based fully convolutional discriminator conditioned on concatenated label maps and images.
- Coarse-to-fine generator: The coarse-to-fine generator combines a global network G1 with a local enhancer G2 to aggregate global and local information.G1 operates at 1024 × 512, while G = {G1, G2} produces 2048 × 1024 images; the networks are trained progressively and then jointly fine-tuned.
- Multi-scale discriminators: Multi-scale discriminators use three identical networks, D1, D2, and D3, to distinguish real and synthesized images at three image scales.The high-resolution images are downsampled by factors of 2 and 4 to form the three-scale image pyramid.
- Improved adversarial loss: The improved adversarial objective adds discriminator feature matching, requiring the generator to match intermediate features and natural statistics across multiple scales.Features are extracted from multiple discriminator layers to stabilize training.
4. Results
The method outperforms prior approaches in semantic fidelity and human-perceived realism, with ablations showing benefits from its loss terms, instance maps, coarse-to-fine generator, and multi-scale discriminators. It also supports interactive editing across scenes, object appearances, facial parts, and datasets.
- Quantitative comparison: Semantic segmentation evaluation shows that our results outperform other methods by a large margin and approach the accuracy of original images.The evaluation measures how well segments predicted from synthesized images match the input label map.
- Human perceptual evaluation: Human studies on Cityscapes use randomized pairwise A/B tests with 5,000 judgments per method under unlimited viewing time.Workers compare images synthesized from the same label map and select which appears more natural.
- Ablation studies: The complete loss is preferred over GAN-only and GAN-plus-feature-matching variants by 68.55% and 58.90%, respectively.The final objective combines GAN loss, discriminator-based feature matching loss, and VGG perceptual loss.
- Ablation studies: Using instance maps achieves a 64.34% preference rate over results without them, improving realism especially around object boundaries.Participants compared highlighted car regions in images generated with and without instance maps.
- Architecture studies: The coarse-to-fine generator outperforms U-Net and CRN architectures, while multi-scale discriminators improve segmentation quality and stabilize adversarial training.The generator comparison evaluates semantic segmentation and human perceptual results; discriminator comparisons also include pairwise A/B tests.
- Interactive editing: Interactive editing changes object labels, car colors, road textures, facial colors, and beards across Cityscapes-style scenes and Helen Face portraits.Examples include replacing trees with buildings and editing facial parts to mimic makeup effects.
5. Discussion and Conclusion
The paper concludes that conditional GANs can synthesize high-resolution photo-realistic images without hand-crafted losses or pre-trained networks. It also extends image-to-image synthesis to diverse outputs and interactive manipulation using suitable training inputs such as instance maps.
- Conclusion: Conditional GANs synthesize high-resolution photo-realistic imagery without hand-crafted losses or pre-trained networks.The approach may benefit domains requiring high-resolution results where pre-trained networks are unavailable, including medical imaging and biology.
- Conclusion: A perceptual loss can slightly improve the results, while the method remains applicable when pre-trained networks are unavailable.The paper specifically identifies medical imaging and biology as potential application domains.
- Extensions: The image-to-image synthesis pipeline produces diverse outputs and supports interactive image manipulation when trained with appropriate input-output pairs, such as instance maps.These capabilities enable users to edit individual facial attributes in real time, including skin color, eyebrows, and beards.
- Extensions: The model learns to stylize different objects without being explicitly told what a texture is, suggesting possible transfer of textures across datasets.The paper gives using textures from one dataset to synthesize images in another as a potential generalization.
A. Training Details
The models were trained from scratch with Adam under a fixed-then-decayed learning-rate schedule and evaluated across several datasets with dataset-specific splits and preprocessing. Inference takes 20–30 milliseconds per 2048 × 1024 image, enabling interactive editing.
- Optimization: Training used Adam from scratch with learning rate 0.0002, held constant for 100 epochs and linearly decayed to zero over the next 100.Weights were initialized from a Gaussian distribution with mean 0 and standard deviation 0.02.
- Hardware: All models were trained on an NVIDIA Quadro M6000 GPU with 24GB GPU memory.
- Inference: 20 ∼30 milliseconds per 2048 × 1024 input image was required for inference on an NVIDIA 1080Ti GPU, supporting interactive image editing.
- Datasets: Cityscapes provided 2975 training and 500 validation images, while NYU Indoor RGBD provided 1200 training and 249 test images.Cityscapes images were 2048 × 1024; NYU images were 561×427.
- Datasets: ADE20K and Helen Face supplied 20210 and 2000 training images, respectively, with preprocessing that resized widths to 512 or images to 1024 × 1024.ADE20K had 2000 test images; Helen Face had 330 test images.
B. Generator Architectures
The generator uses a global generator network and a local enhancer network. Their architectures are specified with convolutional, downsampling, residual, upsampling, and output layers.
- Network structure: The generator consists of a global generator network and a local enhancer network.This two-network design is explicitly stated in the architecture description.
- Layer notation: Reflection padding is used to reduce boundary artifacts, while Rk denotes residual blocks with two 3 × 3 convolutional layers having the same number of filters.The notation also defines c7s1-k and dk as 7 × 7 and 3 × 3 convolutional layers, respectively.
- Global generator: The global network uses c7s1-64,d128,d256,d512,d1024, nine R1024 blocks, u512,u256,u128,u64, and c7s1-3.The notation denotes convolutional, downsampling, residual, upsampling, and final output layers as defined in the passage.
- Local enhancer: The local enhancer uses c7s1-32,d642,R64,R64,R64,u32, and c7s1-3.Its listed sequence contains an initial convolution, downsampling, three residual blocks, upsampling, and an output layer.
C. Discriminator Architectures
The method uses identical 70 × 70 Patch-GAN discriminator architectures for all three discriminators, built from convolutional layers and a final one-dimensional output layer.
- Discriminator architecture: All three discriminators share an identical 70 × 70 Patch-GAN architecture.This architecture follows the Patch-GAN design referenced in the paper.
- Discriminator architecture: Each Ck layer combines 4 × 4 convolution, InstanceNorm, LeakyReLU, and stride 2, with k filters.The first C64 layer omits InstanceNorm, and the LeakyReLU slope is 0.2.
- Discriminator architecture: A final convolution produces a one-dimensional output after the last discriminator layer.
D. Change log
The paper was first released as an initial preprint and later updated for the CVPR camera-ready version with additional edge-to-photo results.
- The initial version was released as a preprint.
- The CVPR camera-ready version added more results for edge-to-photo examples.