Source-linked AI summary
Semi-parametric Image Synthesis
Xiaojuan Qi, Qifeng Chen, Jiaya Jia, Vladlen Koltun
TL;DR
Photographic synthesis needs realistic appearance from semantic layouts while balancing parametric modeling with access to photographic exemplars. SIMS retrieves compatible image segments into a memory bank, aligns and composites them, and uses a deep network to synthesize the output. Across multiple datasets, it produces more realistic images than purely parametric methods, while remaining slower and not trained end-to-end.
Problem
Photographic synthesis must generate realistic images from semantic layouts while parametric models lack access to original photographic content at test time.
Method
SIMS retrieves compatible photographic segments from a memory bank, aligns and composites them, then feeds the canvas and layout to a synthesis network.
Results
Images produced by SIMS were considerably more realistic than purely parametric models across Cityscapes, NYU, and ADE20K.
Takeaways & Limitations
Across five datasets, SIMS images were rated more realistic than Pix2pix in 94% and CRN in 86% of comparisons.
Takeaways & Limitations
The implementation is significantly slower than purely parametric methods, and the pipeline is not trained end-to-end.
Abstract
from arXiv · showhide
We present a semi-parametric approach to photographic image synthesis from semantic layouts. The approach combines the complementary strengths of parametric and nonparametric techniques. The nonparametric component is a memory bank of image segments constructed from a training set of images. Given a novel semantic layout at test time, the memory bank is used to retrieve photographic references that are provided as source material to a deep network. The synthesis is performed by a deep network that draws on the provided photographic material. Experiments on multiple semantic segmentation datasets show that the presented approach yields considerably more realistic images than recent purely parametric techniques. The results are shown in the supplementary video at https://youtu.be/U4Q98lenGLQ
1. Introduction
Photographic image synthesis aims to generate realistic images from rough specifications, but current approaches trade off parametric flexibility against access to photographic references. SIMS combines both sources and produces more realistic results than purely parametric models across several datasets.
- Photographic synthesis from rough specifications could support digital artists and AI systems with visual imagination.
- Parametric models enable end-to-end training but relinquish access to large databases of original photographic content at test time.
- SIMS combines parametric synthesis with a memory bank of photographic segments retrieved for a novel semantic layout.
- The system aligns and composites retrieved segments before a deep network produces the photographic output.
- SIMS images were considerably more realistic than outputs from purely parametric models on Cityscapes, NYU, and ADE20K.
2. Related Work
SIMS differs from predominantly parametric conditional-synthesis methods by retrieving object segments at test time and using them as source material. It extends earlier retrieval and compositing ideas into an automatic deep synthesis pipeline.
- Most related conditional image-synthesis methods use parametric models, including adversarial image-to-image translation and direct convolutional synthesis.
- Figure 1 compares SIMS with Chen and Koltun on coarse semantic layouts from Cityscapes.
- SIMS uses a memory bank of object segments at test time as photographic source material for deep-network synthesis.
- Earlier nonparametric systems retrieved images or object segments for completion, interactive compositing, or scene-collage construction.
- Unlike prior composite-realism and harmonization work, SIMS develops a complete automatic pipeline from semantic layouts.
- The first pipeline stage retrieves compatible segments, aligns them with a spatial transformer, resolves ordering, and composes a canvas.
3. Overview
The pipeline builds a memory bank of semantic segments, retrieves and aligns compatible exemplars for each test layout component, then synthesizes a photographic image from the resulting canvas and layout.
- Training color images and semantic layouts generate a memory bank of connected-component segments associated with semantic classes.
- For each unseen layout component, the system retrieves a compatible segment using shape, location, and context, then aligns it with a spatial transformer.
- Overlapping transformed segments are composited onto a canvas, with an ordering network determining relative front-back relationships.
- The synthesis network takes the canvas and target layout to inpaint regions, harmonize segments, blend boundaries, synthesize shadows, and produce the final image.
- Coarse incomplete layouts are first refined into dense pixelwise layouts before entering the image-synthesis pipeline.
4. External Memory
The external memory stores segmented photographic exemplars and retrieves compatible segments for a novel semantic layout. Retrieved segments are transformed, ordered, and composited into a canvas for synthesis.
- 4.1. Representation: The memory bank stores color images, masks, and surrounding semantic context for segments extracted from training label maps.Context is represented within a bounding box enlarged by 25% in each dimension.
- 4.2. Retrieval: For each test-layout component, the system retrieves a same-class segment using shape overlap and contextual similarity.Mask IoU measures shape overlap, while context IoU compares surrounding semantic layouts.
- 4.4. Canvas: The transformed segments are composited into a canvas whose boundaries are later elided before image synthesis.Boundary elision is applied as part of the canvas preparation process.
- 4.3. Transformation: A transformation network aligns each retrieved segment to its target component through translation, rotation, scaling, and clipping.The network is trained on simulated affine misalignments and cropping, using color-image loss to constrain the transformation.
- 4.4. Canvas: An ordering network resolves overlaps by predicting which adjacent semantic segment should appear in front.Training uses relative depth from dataset-provided depth or stereo data, or approximate depth maps for ADE20K.
5. Image Synthesis
The synthesis network combines a semantic layout with a retrieved-segment canvas to reconstruct and refine a photographic image. Training simulates missing regions, inconsistent appearance, and boundary artifacts so the network learns to correct them.
- 5. Image Synthesis: The synthesis network takes the semantic layout, canvas, and missing-pixel mask as inputs because retrieved canvases contain gaps and visual inconsistencies.It addresses missing regions, illumination and color imbalance, and boundary artifacts jointly rather than using inpainting alone.
- 5.1. Network architecture: The encoder-decoder network uses multi-scale features and skip connections to synthesize progressively finer maps and full-resolution output.The encoder is VGG-19-based, while the decoder is based on the cascaded refinement network.
- 5.2. Training: Training generates simulated canvases by stenciling segments with retrieved masks, transferring colors, and eliding boundaries before recovering the original image.The simulated canvas C′ is constructed from a training image and its layout to mimic test-time artifacts.
- 5.2. Training: Blind randomized A/B tests compare SIMS against Pix2pix and CRN, with each table entry reporting the percentage of judgments favoring SIMS over the corresponding baseline.Chance performance is 50%.
- 5.2. Training: Boundary elision masks 80% of pixels within 0.05h of segment boundaries and also masks nearby exterior pixels to encourage boundary blending and shadow synthesis.Interior pixels are replaced by white pixels, while exterior pixels are replaced by black pixels; the same interior and exterior elision is used at test time.
6. Experiments
Experiments across multiple semantic segmentation datasets evaluate SIMS against parametric baselines using human judgments, semantic consistency, and image statistics. SIMS is consistently judged more realistic, more frequently mistaken for real images, and closer to real-image statistics.
- Experimental setup: Experiments use Cityscapes, NYU, and ADE20K, with Cityscapes evaluated under coarse, fine, and Cityscapes→GTA5 conditions.The Cityscapes→GTA5 condition applies models trained on fine Cityscapes layouts to GTA5 semantic label maps.
- Human evaluation: 94% and 86% of comparisons rated SIMS more realistic than Pix2pix and CRN, respectively, across five datasets.SIMS outperformed the baselines in all reported conditions.
- Human evaluation: After 1 second, SIMS was preferred over real images in 25.2% of Cityscapes-coarse comparisons and 27.8% of Cityscapes-fine comparisons.The corresponding CRN and Pix2pix rates were 4.0% and 3.8% for coarse input, and 15.2% and 1.9% for fine input.
- Semantic consistency: SIMS images were more accurately parsed by PSPNet than Pix2pix or CRN images, with especially large differences on Cityscapes-coarse and ADE20K.This protocol measures both realism and conformance to the input semantic layout.
- Image statistics: SIMS matched the real ADE20K mean power spectrum closely, whereas Pix2pix and CRN showed spurious spectral spikes.The spectra are averaged over ADE20K, with magnitude shown on a logarithmic scale.
- Diversity: SIMS can generate diverse image collections by randomly retrieving among the top k segments for each semantic region and repeating synthesis.The resulting diversity experiments are provided in the supplement.
7. Conclusion
SIMS produces considerably more realistic images than recent purely parametric techniques, while retaining a fallback to parametric synthesis when retrieved photographic material is not useful. The approach remains slower than purely parametric methods, is not trained end-to-end, and leaves video synthesis for future work.
- SIMS produces considerably more realistic images than recent purely parametric techniques.
- If the memory bank is not useful, the network can ignore the canvas and perform parametric synthesis from the input semantic layout.
- The presented pipeline is not trained end-to-end.
- Applying semi-parametric techniques to video synthesis remains an open direction for future work.
- The implementation is significantly slower than purely parametric methods.