Interactive Image Synthesis with Panoptic Layout Generation

1. Introduction

• Scene graphs offer novice-friendly manipulation by requiring object placements rather than professional free-hand semantic-layout drawing.Semantic layouts can produce pleasing results but demand high-precision construction skills.Source: §1 · p. 1
• Grid2Im [1] constructs instance layouts from object masks and bounding boxes before generating images, but requires ground-truth segmentation supervision.This prior pipeline illustrates the instance-layout approach that PLGAN extends.Source: §1 · p. 1
• Instance-layout methods are sensitive to arbitrary object placement because predicted masks may not cover their bounding boxes, leaving empty semantic-layout regions and image artifacts.Training layouts typically cover the image lattice, whereas interactive users can perturb bounding boxes and create uncovered areas.Source: §1 · pp. 1–2
• PLGAN separates amorphous stuff from well-defined things, then fuses their layouts through Instance- and Stuff-Aware Normalization to address missing regions.The stuff branch generates pixel-wise masks covering the image lattice, complementing the instance branch’s per-instance boxes and masks.Source: §1 · p. 2
• PLGAN’s qualitative and quantitative experiments on COCO-Stuff, Visual Genome, and Landscape demonstrate merits over state-of-the-art layout-based approaches.The supplied introduction reports comparisons across all three datasets.Source: §1 · p. 2

Related Work

• Layout-based conditional synthesis supports computer-human interaction by directly leveraging object locations and shapes to produce aesthetically pleasing multi-object images.This motivates layouts as a controllable alternative to less spatially explicit conditions.Source: §2 · p. 2

Instance Layout

• Instance-layout methods assign categories and instance IDs to pixels around independently modeled boxes and shapes, but can leave regions missing in interactive, high-resolution scenes.Scene layouts remain user-friendly because they expose object labels, locations, and interactions.Source: §2 · p. 2

Panoptic Layout

• PLGAN adapts panoptic segmentation by using separate region-based instance layouts for things and dense-pixel semantic layouts for stuff.This transfers panoptic theory from scene understanding into interactive image generation.Source: §2 · p. 2
• PLGAN uses a two-stage scene-to-layout and layout-to-image pipeline, with panoptic layout generation complementing the instance layout’s missing regions.The architecture is trained end-to-end for interactive synthesis.Source: Figure 2 · p. 3

3. Method

• PLGAN maps object inputs through scene-to-layout generation and layout-to-image synthesis, with separate stuff and thing representations in the panoptic layout.Each object is represented by category, center position, and size; the layout-to-image generator produces the final image from the layouts and a latent code.Source: §3 · p. 3

3.1. Panoptic Layout Generation

• Stuff and thing categories are modeled separately because stuff forms amorphous regions while things are countable foreground objects with well-defined shapes.The two layouts are fused and refined using instance-to-instance and instance-to-stuff relations to address missing regions in instance-only layouts.Source: §3.1 · p. 3

3.1.1 Instance Layout Branch

• Instance layout construction predicts each thing’s bounding-box dimensions and mask, then assembles resized masks into an instance layout.The input center position is combined with predicted height and width to obtain each bounding box.Source: §3.1.1 · pp. 3–4

3.1.2 Stuff Layout Branch

• Stuff layout construction begins with coarse square masks, jointly refines them with a conditional generator, and normalizes the result using masked softmax.The coarse masks use each stuff object’s size and center position, while the refinement models intercorrelated stuff regions jointly.Source: §3.1.2 · p. 4

3.2.1 Conditional Image Synthesis

• PLGAN uses CAL2I and related layout-to-image generators, while ISA-Norm fuses panoptic layouts during conditional image synthesis.For 128 × 128 output, an FC layer and five ResBlocks progressively upsample the latent representation, blending layout information through ISA-Norm.Source: §3 · p. 3, §3.2.1 · p. 4

3.2.2 Instance- and Stuff-Aware Normalization

• ISA-Norm replaces instance-only ISLA-Norm because stuff can cover the scene and overlap extensively with instances in panoptic layouts.It constructs spatial scale and shift parameters from separately embedded stuff and thing information, using a foreground mask and guided filtering before fusion.Source: §3.2.2 · pp. 4–5

3.3. Model Objectives

• PLGAN training combines bounding-box MSE, image and object adversarial losses, perceptual and reconstruction losses, and appearance adversarial loss.The appearance loss uses Gram matrices to measure spatial similarity between object features, while the perceptual loss uses VGG-19 activations.Source: §3.3 · p. 5

3.4. Implementation Details

• The model is trained with Adam at learning rate 10^-4, batch size 128, and 200 epochs on NVIDIA Tesla V100 hardware.Inference is tested on a Huawei Atlas workstation with an Ascend AI accelerator and CANN.Source: §3.4 · p. 5

4. Experiments

• PLGAN is evaluated on COCO-Stuff, Visual Genome, and Landscape using visual and quantitative comparisons, including IS, FID, CAS, and Coverage.The experiments also include ablations of PLG and ISA-Norm.Source: §4 · pp. 5–7

4.1. Qualitative Results

• Perturbed bounding boxes cause artifacts for instance-layout models, whereas PLGAN’s stuff processing fills regions between boxes and image borders.The qualitative comparison covers 128 × 128 and 256 × 256 resolutions and also highlights gaps caused by mismatched instance masks.Source: §4.1 · p. 7

4.2. Quantitative Results

• PLGAN achieves its clearest IS and FID advantage under Pert1 BBox and Pert2 BBox, while its CAS is higher across tested resolutions with GT BBox input.The comparison uses GT BBox and two perturbation ranges, [−0.3, 0.3] and [−0.5, 0.5].Source: §4.2 · pp. 7–8, Table 1 · p. 7
• As perturbation increases from 0 to 0.5, PLGAN remains the best overall performer in IS and FID and maintains 100% COV while instance baselines lose coverage.The plotted metrics are IS, FID, and COV under 128 × 128 resolution.Source: §4.2 · p. 8, Figure 7 · p. 8

4.3. Ablation Studies

• The PLG module improves instance-layout baselines, while combining PLG with ISA-Norm yields the highest scores across ablation metrics and preserves real-time performance.PLG alone can degrade image quality, whereas ISA-Norm remedies that degradation for panoptic layouts.Source: §4.3 · p. 8, §4.2 · p. 8, Table 2 · p. 8
• Using only a stuff or only a thing branch produces inferior metrics to the combined panoptic layout approach.This ablation supports treating stuff and thing objects through separate branches and combining them.Source: §4.3 · p. 8, Table 3 · p. 8

5. Conclusion

• PLGAN separates instance and stuff layout branches to improve robustness and eliminate missing regions in interactive scene-to-image synthesis.The conclusion reports evidence across COCO-Stuff, Visual Genome, and Landscape against state-of-the-art methods.Source: §5 · p. 8

A.1. Qualitative Results on Visual Genome

• Visual Genome results compare LostGAN-V1, CAL2I, and PLGAN with perturbed bounding boxes, alongside synthesized images and panoptic layouts.The visual comparisons use the Visual Genome dataset and include 128 × 128 results.Source: §A.1 · p. 11

A.2. Quantitative Results on Visual Genome

• Visual Genome models are quantitatively compared using Inception Score, FID, and CAS.The appendix describes this comparison as analogous to the COCO-Stuff results.Source: §A.2 · p. 11

• PLGAN and Grid2Im are visually compared on the Landscape dataset.Source: §A.3 · p. 11

A.4. Quantitative Results on Landscape

• PLGAN outperforms Grid2Im [1] on all metrics for Landscape, where every object is stuff.The all-stuff composition makes the performance difference more apparent.Source: §A.4 · p. 11

A.5. Robustness to Perturbed BBoxes

• PLGAN is reported as the most robust model as perturbation ranges vary on COCO-Stuff at 256 × 256 resolution.The comparison includes Grid2Im [1] and LostGAN-V2 [35] using IS, FID, and Coverage curves.Source: §A.5 · p. 11, Figure 11 · p. 15

A.6. User Study

• A 600-vote user study compares realism for 100 grouped samples generated by Grid2Im, LostGAN, CAL2I, and PLGAN.Each vote selects one of two images from the same group, with aggregate scores reported in Table 6.Source: §A.6 · p. 11, Table 6 · p. 14

B. Guide Filter

• The Guided Filter maps image features to a three-channel guide, filters instance layouts, predicts affine parameters, and computes a refined layout.The workflow uses convolutional layers, prescribed kernels, mean and covariance operations, and the relation b = L_i^T h − A ⊙ Xg.Source: B. Guide Filter · p. 11