PaperlayerView exact revision on arXiv

Source-linked AI summary

High-Resolution Complex Scene Synthesis with Transformers

Manuel Jahn, Robin Rombach, Björn Ommer

arXiv:2105.06458v1cs.CV

TL;DR

High-resolution synthesis of complex scenes from coarse layouts remains challenging. The method uses adversarially trained discrete image representations and a layout-conditioned autoregressive transformer trained with likelihood learning. The method improves prior state-of-the-art FID by 19–53% at 256×256 px and provides first layout-guided synthesis experiments on Open Images.

  • Problem

    High-resolution synthesis of complex scenes from coarse layouts remains challenging.

  • Method

    The method uses adversarially trained discrete image representations and a layout-conditioned autoregressive transformer trained with likelihood learning.

  • Results

    The method improves prior state-of-the-art FID by 19–53% at 256×256 px and provides first layout-guided synthesis experiments on Open Images.

  • Takeaways & Limitations

    A conceptually simple latent-space transformer can outperform specialized systems and generalize from Open Images to comparable zero-shot COCO performance.

  • Takeaways & Limitations

    Autoregressive sampling is slower than parallel methods, and transformers are data-demanding.

Abstract

from arXiv · show

The use of coarse-grained layouts for controllable synthesis of complex scene images via deep generative models has recently gained popularity. However, results of current approaches still fall short of their promise of high-resolution synthesis. We hypothesize that this is mostly due to the highly engineered nature of these approaches which often rely on auxiliary losses and intermediate steps such as mask generators. In this note, we present an orthogonal approach to this task, where the generative model is based on pure likelihood training without additional objectives. To do so, we first optimize a powerful compression model with adversarial training which learns to reconstruct its inputs via a discrete latent bottleneck and thereby effectively strips the latent representation of high-frequency details such as texture. Subsequently, we train an autoregressive transformer model to learn the distribution of the discrete image representations conditioned on a tokenized version of the layouts. Our experiments show that the resulting system is able to synthesize high-quality images consistent with the given layouts. In particular, we improve the state-of-the-art FID score on COCO-Stuff and on Visual Genome by up to 19% and 53% and demonstrate the synthesis of images up to 512 x 512 px on COCO and Open Images.

1. Introduction

High-resolution synthesis of complex scenes from coarse layouts remains challenging because models must capture object appearance and interactions. The paper proposes a simpler latent-space transformer approach using pure autoregressive likelihood training without auxiliary objectives.

  • High-resolution synthesis of complex scenes from coarse layouts remains challenging because models must capture object appearance and interactions.
  • The approach builds on latent-space autoregressive transformers and extends them to high-resolution scene synthesis using pure likelihood training without additional objectives.
  • Related Work: Transformers model global interactions through attention rather than locality-biased convolutional operations.
  • VQGAN-style adversarial reconstruction provides a compressed discrete latent space suitable for training an autoregressive likelihood model.

2. Method

The method compresses images into discrete latent tokens and trains an autoregressive transformer to model those tokens conditioned on tokenized coarse layouts. Layouts encode object classes and bounding-box positions using fixed-length sequences.

  • Efficient Training in Latent Space: Adversarially trained vector quantization creates a low-dimensional discrete codebook so the transformer models large-scale structure instead of short-range pixel interactions.
  • Efficient Training in Latent Space: The transformer learns the conditional likelihood of discrete image tokens through autoregressive next-element prediction.
  • Efficient Training in Latent Space: Conditioning information must be tokenized, allowing the same objective to incorporate labels, text, or another model’s representation.
  • Encoding Coarse Layouts: Each layout object is encoded by its class and top-left and bottom-right coordinates, with positions mapped to a virtual grid.
  • Encoding Coarse Layouts: Layouts use a fixed conditional sequence length, padding sequences with null tokens when fewer than the maximum number of objects are present.

3. Experiments

Experiments evaluate layout-to-image synthesis on COCO, Visual Genome, and Open Images across increasing scene complexity and resolution. The model reaches competitive or improved FID-based performance, including 512-pixel synthesis and transfer from Open Images to COCO.

  • Results on Complex Scene Synthesis: The experiments cover COCO and Visual Genome at up to 256 pixels, then extend to 512-pixel synthesis and Open Images.
  • Results on Complex Scene Synthesis: The study also evaluates zero-shot transfer by rendering COCO validation layouts with a model trained exclusively on Open Images.
  • Results on Complex Scene Synthesis: Visual Genome reaches performance comparable to previous methods with basic augmentation, while smaller COCO requires random cropping for competitive results.
  • Results on Complex Scene Synthesis: Table 3 reports 13–60% lower FID than [27] on a harder 2–30-object setting, although object-count differences limit comparability.
  • Results on Complex Scene Synthesis: The model outperforms previous methods by up to 18 FID points at 256 pixels when trained with full augmentation and a COCO data superset.

4. Conclusion

The paper concludes that adversarially learned discrete latents let a simple likelihood-trained transformer outperform specialized systems and scale to high-resolution scene synthesis. Training on Open Images can also yield comparable COCO performance without COCO training.

  • Adversarially trained discrete latents provide high-fidelity reconstructions and a suitable space for maximum-likelihood modeling of large-scale structure.
  • Conditioned on tokenized coarse layouts and without auxiliary losses, the method outperforms prior state-of-the-art systems and supports high-resolution generation.
  • A model trained on Open Images achieves comparable performance on COCO in a zero-shot setting.

A.1. First stage: Details on VQGAN

The first stage uses a VQGAN compression model trained on 256×256 crops, producing a compact discrete representation for high-resolution synthesis.

  • A.1. First stage: Details on VQGAN: An 8192-token codebook represents images as 16×16 token grids after four downsamplings.Each codebook vector has dimensionality 256.
  • A.1. First stage: Details on VQGAN: The VQGAN trained for 117 epochs on full-COCO 256×256 crops achieves a reconstruction FID of 11.0 on 5k validation examples.

A.2. Second stage: Details on Transformer

The second stage uses GPT-2-style autoregressive transformers, scaling model capacity for broader and more difficult scene-generation tasks.

  • A.2. Second stage: Details on Transformer: The comparison model uses approximately 320 million parameters across 24 layers with 16 attention heads and 1024-dimensional embeddings.Its dropout rate is 10%.
  • A.2. Second stage: Details on Transformer: A larger 1.1-billion-parameter transformer is used for the broader and more difficult tasks in Table 3.This model has 36 layers and 1536-dimensional embeddings.

A.3. Details on Datasets

The evaluation spans filtered COCO and Visual Genome benchmarks plus the much larger Open Images dataset, whose object-focused annotations complement COCO’s stuff categories.

  • A.3. Details on Datasets: The COCO segmentation subset contains 50K images with bounding boxes for 80 object and 91 stuff classes, filtered to scenes containing 3 to 8 sufficiently large objects.The split contains 24,972 training, 1,024 validation, and 2,048 test images.
  • A.3. Details on Datasets: Full COCO provides 164K images, with 74,121 training images under the 3-to-8-object filter and 112K under the broader 2-to-30-object filter.Validation uses the same 2,048 test images as the segmentation setting for comparability.
  • A.3. Details on Datasets: Visual Genome contributes 62,565 training images under the standard filter and 80K training images in the broadened setting, covering 178 object classes.Its validation and test sets contain 5,506 and 5,088 images, respectively.
  • A.3. Details on Datasets: Open Images V6 contains 1.9 million images, 16 million bounding boxes, and 600 object classes, making it roughly eleven times larger than COCO.The dataset is filtered toward classes with more than 1,000 bounding boxes and expanded for compatibility with COCO.
  • A.3. Details on Datasets: Visual Genome and Open Images emphasize objects, unlike COCO, and lack COCO’s texture and material categories such as straw, sand, and sky.

A.4. Details on Layout Encoding

The layout encoding reduces each object representation from five tokens to three by placing objects on a virtual raster-scanned grid, limiting sequence length for attention.

  • A.4. Details on Layout Encoding: A virtual image grid encodes layout positions continuously in raster-scan order, allowing coordinates to be recovered from each position number.The x-position is computed using a modulo operation with the number of grid columns, while the y-position follows from the grid index.
  • A.4. Details on Layout Encoding: The three-token encoding is chosen because attention cost grows quadratically with sequence length.

A.5. A Note on Evaluation

The evaluation uses fixed test splits and standardized preprocessing to support comparisons across datasets and models.

  • Evaluation protocol: FID and SceneFID are computed with Torch-Fidelity on 2048 unaugmented COCO test images and 5088 Visual Genome test images.COCO evaluation uses the Segmentation Challenge split, while Visual Genome uses its test split.
  • Evaluation protocol: COCO and Visual Genome test images are resized to squares without test-time augmentation, even when training uses cropped or undistorted images.The COCO and Visual Genome evaluation settings retain this preprocessing for comparability.
  • Evaluation protocol: For the three-dataset comparison, all models are evaluated on 2048 test images using center crops because training used undistorted random crops.This setting differs slightly from the COCO and Visual Genome evaluations.

A.6. Limitations

The reported system produces high-resolution, layout-consistent images with strong texture quality, while retaining practical limitations in speed, data requirements, and some object details.

  • Limitations: Autoregressive sampling is slower than parallel methods, creating a quality–speed trade-off for transformer-based generation.The paper identifies sampling speed as a practical cost of the approach.
  • Limitations: Transformers are data-demanding, although random crops on COCO provide gains roughly equivalent to a 250% increase in training data.The reported augmentation effect indicates that preprocessing can materially affect data efficiency.
  • Zero-shot generation: The model generalizes from Open Images to COCO layouts with higher diversity, but the Open Images model is not fully converged.The COCO-specialized model instead presents higher visual quality in the zero-shot comparison.
  • High-resolution synthesis: At 512×512 pixels, the system demonstrates high-quality scene synthesis, while comparisons report that producing the right amount of legs remains an open challenge.The model is also described as improving consistency and sharp textures relative to LostGAN-v2.
Loading 2105.06458v1…