Source-linked AI summary
Photographic Image Synthesis with Cascaded Refinement Networks
Qifeng Chen, Vladlen Koltun
TL;DR
The paper addresses photographic image synthesis from semantic layouts, where a layout must determine a plausible scene appearance. It uses a single feedforward convolutional network with cascaded refinement and direct supervised regression, rather than adversarial training. The approach reaches 2-megapixel resolution and is judged more realistic than alternative approaches, while remaining distinguishable from real HD images.
Problem
The paper asks whether computational models can synthesize photorealistic images that conform to novel semantic layouts.
Method
A single feedforward convolutional network maps pixelwise semantic layouts to photographs through end-to-end supervised regression and cascaded multi-resolution refinement.
Results
The model synthesizes images at 2 megapixels (1024×2048), and perceptual experiments find them significantly more realistic than alternative approaches.
Takeaways & Limitations
Direct supervised synthesis without adversarial training can produce photographic images from semantic layouts at high resolution.
Takeaways & Limitations
The synthesized images are significantly more realistic than prior work but are not indistinguishable from real HD images.
Abstract
from arXiv · showhide
We present an approach to synthesizing photographic images conditioned on semantic layouts. Given a semantic label map, our approach produces an image with photographic appearance that conforms to the input layout. The approach thus functions as a rendering engine that takes a two-dimensional semantic specification of the scene and produces a corresponding photographic image. Unlike recent and contemporaneous work, our approach does not rely on adversarial training. We show that photographic images can be synthesized from semantic layouts by a single feedforward network with appropriate structure, trained end-to-end with a direct regression objective. The presented approach scales seamlessly to high resolutions; we demonstrate this by synthesizing photographic images at 2-megapixel resolution, the full resolution of our training data. Extensive perceptual experiments on datasets of outdoor and indoor scenes demonstrate that images synthesized by the presented approach are considerably more realistic than alternative approaches. The results are shown in the supplementary video at https://youtu.be/0fhUJT21-bs
1. Introduction
The paper asks whether computational models can synthesize photorealistic images from semantic layouts. It proposes a supervised convolutional approach and reports realistic outputs without adversarial training, including at high resolution.
- The central question is whether an artificial system can turn a novel semantic layout into a conforming photographic image.
- Such synthesis could serve as an alternative rendering route that avoids detailed 3D geometry, surface reflectance specification, and intensive light-transport simulation.
- The model is a supervised convolutional network trained on paired photographs and pixelwise semantic layouts to synthesize photographs rather than infer layouts.
- A single feedforward convolutional network trained with a regression loss synthesizes photographic images without adversarial training and scales to 2 megapixels (1024×2048).
- Perceptual experiments on synthesized images found them significantly more realistic than images from alternative approaches.
2. Related Work
Related work spans adversarial, direct feedforward, and other conditional image-synthesis approaches. The paper distinguishes its setting by synthesizing high-resolution photographic scenes from semantic layouts alone with one end-to-end network and no adversarial training.
- Prior multi-scale GAN work improved image smoothness and resolution, while this paper trains one end-to-end model with direct synthesis and no adversarial training.
- Earlier GAN research required architectural modifications and heuristics to scale adversarial training and address its instability.
- Direct feedforward regression had been used to render images of 3D models, whereas this work targets sharper, higher-resolution scenes without 3D models.
- Composite-loss approaches combine regression features with a GAN loss for image synthesis tasks.
- Compared with Isola et al., the paper uses a simpler direct formulation for the same semantic-layout setting and reports more realistic results.
- Other conditional synthesis studies generate low-resolution images from text, attributes, spatial constraints, or factorized scene representations, often using GANs.
- Video synthesis commonly starts from an available color image, unlike this setting, which must synthesize scene appearance without such initialization.
- Inpainting, superresolution, novel-view synthesis, and interactive manipulation receive photographic content as input, whereas this work starts from semantic layouts alone.
3. Method
The method maps pixelwise semantic layouts to photographic images using cascaded multi-resolution refinement, perceptual supervision, and optional diversity mechanisms. Its architecture progressively upsamples features while retaining layout information at each resolution.
- 3.1. Preliminaries: A semantic layout is a one-hot tensor over pixel resolution m×n and c semantic classes, and the goal is to map it to a conforming color image.
- 3.1. Preliminaries: Global coordination, multi-resolution refinement, and sufficient model capacity are identified as important for photorealistic synthesis.
- 3.2. Architecture: The Cascaded Refinement Network begins at 4×8 resolution and doubles resolution between successive refinement modules.
- 3.2. Architecture: The first module processes a downsampled layout, while later modules concatenate the layout with an upsampled preceding feature layer.
- 3.2. Architecture: Each module uses an input, intermediate, and output feature layer; bilinear upsampling replaces upconvolutions to avoid characteristic artifacts.
- 3.2. Architecture: The final module linearly projects its feature output to a three-channel color image; the high-resolution Cityscapes model uses 9 modules to reach 1024×2048.
- 3.3. Training: Training pairs a semantic layout with a corresponding reference image in an underconstrained one-to-many inverse problem.
- 3.3. Training: A perceptual loss matches activations from low- and high-level layers of VGG-19, guiding both fine details and global part arrangement.
4. Baselines
The paper evaluates alternative synthesis systems as baselines, varying adversarial objectives, network architecture, and feature-matching loss while controlling other factors where possible.
- GAN and semantic segmentation: The first baseline combines a GAN objective with semantic segmentation loss so synthesized images both fool a discriminator and preserve the input layout.A pretrained segmentation network evaluates whether the generated image reconstructs the input semantic map.
- Full-resolution network: The full-resolution baseline uses a feedforward convolutional architecture that retains high resolution in intermediate layers while applying the CRN loss.Its architecture is varied to test whether preserving high resolution benefits photorealistic synthesis.
- Encoder-decoder: The encoder-decoder baseline uses a u-net trained with the CRN loss, isolating the effect of architecture while holding loss and training procedure fixed.This comparison specifically evaluates CRN architecture against an alternative network under matched training conditions.
- Image-space loss: The image-space baseline keeps the CRN architecture but uses only the first feature layer, Φ0, corresponding to image color, in the loss.This controls for the CRN’s feature-matching loss while changing the loss representation.
- Image-to-image translation: The final baseline is a contemporaneous conditional GAN image-to-image translation system whose generator uses an encoder-decoder architecture.Its GAN design follows prior work and represents the dominant image-synthesis approach described by the paper.
5. Experiments
The experiments evaluate realism with human pairwise comparisons across Cityscapes and NYU, including unlimited- and time-limited viewing. The approach is compared against baselines and qualitatively assessed for high-resolution synthesis and output diversity.
- Experimental procedure: Human observers compare images generated for the same label map by two approaches, selecting the more realistic image in Amazon Mechanical Turk A/B tests.Jobs contain roughly 100 pairwise comparisons and sentinel pairs for attention checks.
- Experimental procedure: The study uses unlimited-viewing and randomly time-limited comparisons, with the latter testing how quickly relative realism can be established.Time-limited pairs are shown for durations between 1/8 and 8 seconds.
- Primary experiments: On Cityscapes, the presented approach was rated more realistic than four alternative approaches, with all results statistically significant at p < 10^-3.The comparisons include controls for both loss and architecture.
- Time-limited comparisons: At 1/8 second, CRN images were rated on par with real images and more realistic than Pix2pix images, with Real>CRN at 52.6% and CRN>Pix2pix at 73.4%.The corresponding Real>Pix2pix rate was 72.5%.
- Time-limited comparisons: At 500 milliseconds, CRN>Pix2pix reached 87.4%, while Real>Pix2pix reached 95.1% and Real>CRN reached 64.2%.Over time, CRN>Pix2pix rose above 90%.
- NYU dataset: On NYU, the presented approach was again rated consistently more realistic than baselines, despite the dataset’s smaller size and lower resolution.All results were statistically significant at p < 10^-3.
- Diversity loss: With diversity loss, nine-image output collections produced different appearances for a given semantic layout.The experiments sample pairs from collections synthesized for NYU validation layouts.
6. Conclusion
The paper presents a direct, simpler approach to synthesizing photographic images from pixelwise semantic layouts. Its results are more realistic than prior state-of-the-art methods, but remain distinguishable from real HD images.
- The approach synthesizes photographs from pixelwise semantic layouts using an end-to-end convolutional network with a regression loss.
- The direct approach is considerably simpler than contemporaneous work and produces much more realistic results.
- The synthesized results are significantly more realistic than prior state-of-the-art results but are not indistinguishable from real HD images.