PaperlayerView exact revision on arXiv

Source-linked AI summary

Illiterate DALL-E Learns to Compose

Gautam Singh, Fei Deng, Sungjin Ahn

arXiv:2110.11405v3cs.CVcs.LG

TL;DR

DALL·E provides systematic zero-shot image composition but relies on text-image supervision, while object-centric models learn from images yet have limited zero-shot reconfiguration. SLATE addresses this gap with image-derived slots and a slot-conditioned Image GPT decoder, achieving significant improvements in image generation and comparable or better slot structure.

  • Problem

    DALL·E relies on text to provide composable structure, while image-only object-centric models have limited systematic generalization for zero-shot generation.

  • Method

    SLATE is a slot-based autoencoder that learns object-centric representations from images and decodes them with a slot-conditioned Image GPT transformer.

  • Results

    SLATE significantly improves in-distribution and out-of-distribution image generation while producing qualitatively comparable or better slot-attention structure than mixture-decoder models.

  • Takeaways & Limitations

    Text-free object-centric generation can combine reusable image-derived slots with globally consistent novel compositions.

  • Takeaways & Limitations

    The paper suggests future object-centric models should investigate how to combine SLATE’s framework with pixel-mixture decoders.

Abstract

from arXiv · show

Although DALL-E has shown an impressive ability of composition-based systematic generalization in image generation, it requires the dataset of text-image pairs and the compositionality is provided by the text. In contrast, object-centric representation models like the Slot Attention model learn composable representations without the text prompt. However, unlike DALL-E its ability to systematically generalize for zero-shot generation is significantly limited. In this paper, we propose a simple but novel slot-based autoencoding architecture, called SLATE, for combining the best of both worlds: learning object-centric representations that allows systematic generalization in zero-shot image generation without text. As such, this model can also be seen as an illiterate DALL-E model. Unlike the pixel-mixture decoders of existing object-centric representation models, we propose to use the Image GPT decoder conditioned on the slots for capturing complex interactions among the slots and pixels. In experiments, we show that this simple and easy-to-implement architecture not requiring a text prompt achieves significant improvement in in-distribution and out-of-distribution (zero-shot) image generation and qualitatively comparable or better slot-attention structure than the models based on mixture decoders.

1 INTRODUCTION

The paper asks whether image-only object-centric representations can support the systematic, zero-shot composition achieved by text-guided models. It proposes SLATE to combine image-derived slots with a transformer decoder for text-free generation.

  • Unsupervised compositional representations support perceiving scenes as component structures and imagining novel scenes beyond the training distribution.
  • DALL·E achieves zero-shot image generation from unfamiliar text prompts, but the prompt already supplies the composable structure.
  • The central challenge is extracting composable representations from images alone to replace the role of word tokens in systematic image generation.
  • Object-centric models learn slots from images, but their slot decoding significantly limits systematic generalization to arbitrary slot reconfiguration.
  • SLATE combines image-only object-centric learning with a transformer decoder to improve composition-based systematic generalization without text supervision.
  • The paper presents SLATE as a simpler text-free DALL·E that improves systematic generalization in object-centric representation models.

2 PRELIMINARIES

Object-centric autoencoders reconstruct images by decoding and composing slot representations, commonly through pixel-wise mixtures. The paper identifies decoder-capacity and pixel-independence problems and contrasts them with autoregressive transformer decoding.

  • Object-centric autoencoders encode an image into slot vectors and reconstruct it by composing each slot’s decoded representation.
  • Pixel-mixture decoders generate a slot image and alpha-mask for each slot, then combine them through a pixel-wise weighted mean.
  • LIMITATIONS OF PIXEL-MIXTURE DECODERS: Pixel-mixture decoders face a slot-decoding dilemma: weak decoders encourage object-focused slots but can blur image details, while expressive decoders can undermine disentanglement.
  • LIMITATIONS OF PIXEL-MIXTURE DECODERS: Pixel independence prevents slots from interacting with other slots and pixels, limiting arbitrary reconfiguration and global semantic consistency.
  • IMAGE GPT AND DALL·E: Image GPT autoregressively models image-token sequences, so generated pixels depend on preceding tokens rather than remaining independent.
  • IMAGE GPT AND DALL·E: DALL·E conditions Image GPT on text tokens for globally consistent zero-shot generation, but requires text-image supervision instead of learning compositional structure from images.

3 SLATE: SLOT ATTENTION TRANSFORMER

SLATE learns object slots from images alone and uses a transformer decoder to reconstruct tokenized images, while a clustered slot library supports compositional generation.

  • 3 SLATE: SLOT ATTENTION TRANSFORMER: SLATE uses a transformer image decoder conditioned on object slots to provide global semantic consistency without text.The decoder replaces text prompts with slot prompts drawn from a visual concept library.
  • 3 SLATE: SLOT ATTENTION TRANSFORMER: Input images are downscaled and converted into discrete patch tokens with a DVAE before transformer training.The DVAE partitions images into K × K patches, samples relaxed categorical token encodings, and reconstructs patches with an MSE objective.
  • 3 SLATE: SLOT ATTENTION TRANSFORMER: The encoder maps patch-token embeddings with positional information to N object slots and N attention maps using Slot Attention.Each patch token is dictionary-embedded, combined with a learned positional embedding, and processed by the slot-attention encoder.
  • 3 SLATE: SLOT ATTENTION TRANSFORMER: During reconstruction, a transformer predicts DVAE tokens from the slots, and the DVAE decoder converts those tokens back into image patches.The reconstructed patches together form the output image.
  • 3 SLATE: SLOT ATTENTION TRANSFORMER: The transformer is trained by minimizing cross-entropy for each image token conditioned on preceding tokens and the slot set.The complete objective combines the transformer token loss LST with the DVAE reconstruction loss LDVAE, and all modules are trained jointly.
  • 3 SLATE: SLOT ATTENTION TRANSFORMER: SLATE builds a reusable visual concept library by clustering slots from training images and composing images from sampled cluster representatives.Cosine similarity clusters slots by appearance, while attention-map IOU can be used when object position matters more.

4 RELATED WORK

Related work covers object-centric and GAN-based compositional generation, while distinguishing SLATE through unsupervised object discovery and transformer-based comparison with Slot Attention.

  • 4 RELATED WORK: Self-supervised object-centric approaches provide the broader research context for compositional generation from images.The cited approaches include methods for object-centric representation learning and compositional generation without text prompts.
  • 4 RELATED WORK: Prior GAN-based methods use manually supplied patch collages or specialized object-discovery procedures, unlike SLATE.Other approaches randomly redraw or jitter foregrounds and train with adversarial losses.
  • 4 RELATED WORK: Table 1 compares Slot Attention and SLATE on compositional-generation FID and preference votes, plus reconstruction MSE and FID.A vote percentage above 50% indicates preference for SLATE over the Slot Attention baseline.
  • 4 RELATED WORK: Some GAN compositional-generation methods require supervision for bounding boxes or keypoints, whereas SLATE does not rely on those annotations.The cited methods render scenes through object-centric or keypoint-based pathways.

5 EXPERIMENTS

Experiments compare SLATE’s transformer decoder with a Slot Attention mixture decoder across compositional generation, reconstruction, and out-of-distribution settings using raw images without annotations. SLATE produces more realistic compositional generations, reuses slots effectively, and improves image quality while preserving object-focused attention masks.

  • Experimental Setup: SLATE is evaluated against a Slot Attention mixture-decoder baseline on seven composable-object datasets using only raw images.The evaluation covers novel images from arbitrary slot configurations, reconstruction, and out-of-distribution generation.
  • Compositional Image Generation: SLATE’s compositional generations are significantly more realistic than those from the mixture decoder and accurately reuse known slots.The comparison uses discriminator training curves and qualitative compositions from arbitrary slot prompts.
  • Compositional Image Generation: SLATE performs zero-shot composition with novel slot inputs, including object-count changes and arbitrary block configurations, whereas the mixture decoder fails on arbitrary slot composition.The tests include CLEVR scenes with 1–2 or 7–8 objects and Shapestacks towers composed from concept-library blocks.
  • Compositional Image Generation: SLATE renders fine details such as floor textures and complex facial features better, while its attention masks effectively focus on object regions in textured images.Slot Attention can fail to segment digits or merge similarly colored hair and background regions.
  • Reconstruction Quality: SLATE outperforms Slot Attention in FID on all datasets, with larger improvements as visual complexity increases, while achieving comparable MSE overall.The baseline has better MSE on 3D Shapes, CLEVR, and CelebA, but substantially poorer novel-slot rendering.

6 DISCUSSION

SLATE enables text-free zero-shot imagination by learning reusable slot representations from images alone. Its transformer decoder improves slot reconfigurability, while future work could formalize concept-library growth and slot-level sampling.

  • SLATE learns slot representations from images alone to support text-free zero-shot imagination.
  • Its transformer decoder significantly improves slot reconfigurability over pixel-mixture decoders for zero-shot imagination.
  • The decoder requests abstract object-level what and where information from slots, then models appearance, occlusions, shadows, reflections, and transparency.
  • Future work could formalize reusable concept libraries through online or Bayesian non-parametric clustering and learn slot-level priors for scene sampling.

ETHICS STATEMENT

The current model is not realistic enough to deceive humans, but scaled-up future versions could have that potential. These societal consequences are described as not imminent.

  • The current model does not generate images realistic enough to deceive humans.
  • Scaling up model size and training data could create potential for deceptive images, although these consequences are not imminent.

A.1 ARCHITECTURE

SLATE converts image patches into discrete tokens, infers slots from token embeddings, and reconstructs images through two pathways. CNN processing is added for textured datasets before slot inference.

  • SLATE splits images into patches and uses a DVAE encoder to represent each patch as a discrete token.
  • A DVAE decoder reconstructs each patch from its token using a simple MSE reconstruction loss.
  • The encoder infers slots and attention maps from token embeddings using Slot Attention.
  • For textured datasets, a CNN gives patch embeddings a larger receptive field before positional embeddings are supplied to Slot Attention.
  • The CNN architecture for Textured-MNIST, CelebA, and CLEVRTex uses 192 channels, matching the DVAE embedding size.

A.4 MULTI-HEADED SLOT ATTENTION

Multi-Headed Slot Attention improves slot expressiveness for complex objects by allowing each slot to attend to multiple object parts. Its benefits depend on the decoder and object complexity, while the architecture is evaluated through ablations and implementation choices.

  • Motivation and mechanism: Multi-Headed Slot Attention lets each slot attend to different parts of the same complex object and concatenate those features for richer representations.
  • Results: On Bitmoji, using 4 heads significantly improves reconstruction MSE compared with standard one-head Slot Attention.
  • Results: With a mixture decoder, multiple heads do not improve reconstruction because the Spatial Broadcast decoder is the performance bottleneck.
  • Limitations: On simple-object datasets, multiple heads can harm disentanglement by encouraging one slot to collect information about multiple objects.
  • Ablation: In 3D Shapes, transformer-decoder models outperform mixture-decoder models in FID, indicating the transformer decoder is the main generation-quality driver in this ablation.
  • Ablation and implementation: The architecture compares VQ input plus Transformer decoding against CNN input and mixture-decoder substitutions, with additional training hyperparameters and regularization reported.

D ADDITIONAL QUALITATIVE RESULTS

Additional qualitative results show that slot representations cluster by object semantics and support object replacement, out-of-distribution compositions, and qualitative comparisons across datasets.

  • Slots from 3D Shapes, Textured MNIST, and Bitmoji form clusters associated with object concepts or classes.The Bitmoji results state that same-class objects tend to have similar representations and automatically form clusters.
  • Replacing a slot with another slot from the same concept library enables object editing in Bitmoji.The replacement procedure substitutes a specific slot with an arbitrary slot representing the same concept.
  • Opposite-gender slot replacements provide examples of out-of-distribution compositions in Bitmoji.The qualitative hair-replacement results use source slots from images with the opposite gender.
  • Qualitative compositional-generation results are reported for Bitmoji, 3D Shapes, Shapestacks, TexturedMNIST, CLEVRTex, and CLEVR.The figures include comparisons with Slot Attention for Bitmoji and CLEVR, and additional comparisons across several datasets.
  • Out-of-distribution compositional-generation results are shown for Shapestacks from SLATE and Slot Attention.

E.1 STRONG DECODER IN MIXTURE LIKELIHOOD MODELS MAKE SLOTS CAPTURE MULTIPLE OBJECTS

The appendix analyzes why mixture decoders trade off reconstruction capacity against object disentanglement and describes how SLATE’s transformer decoder avoids this trade-off while reducing color bias.

  • Spatial Broadcast decoding produces finer image detail only slowly because its coordinate-to-RGB mapping begins as a constant or linear function.This early-training behavior biases object-component decoders toward simple image segments.
  • 37: A more powerful mixture decoder can cause slots to capture multiple objects or even the entire scene.The analysis explains that a weak decoder encourages disentanglement, whereas a powerful decoder removes that incentive.
  • SLATE attention maps localize individual objects or meaningful segments across Shapestacks, Bitmoji, CLEVR, CLEVRTex, and 3D Shapes.The figures report localization on blocks, face segments, objects, and scene components such as walls and floors.
  • SLATE’s transformer decoder is presented as eliminating the historical trade-off between decoding capacity and object disentanglement.The model is described as detecting objects without supervision while rendering fine details during decoding.
  • Color bias can split multicolored objects across slots or merge similarly colored objects into one slot.These effects are demonstrated in CelebA and Textured-MNIST attention maps.
  • Ablations indicate that discrete input representations from DVAE and a transformer decoder can work synergistically for object discovery.

F ADDITIONAL EXPERIMENT DETAILS

Additional experiments describe position-based slot prompting, introduce CLEVR-Mirror for testing global consistency, and construct a dataset using MNIST digits and texture data.

  • For CLEVR and Shapestacks, slot prompts are organized by object position using attention-map intersection-over-union matching.Slots are assigned to library regions with similar patch locations, and prompts are sampled from selected regions.
  • CLEVR-Mirror extends CLEVR with a mirror to test whether models learn global consistency between local scene components.The dataset requires relational reasoning about reflected-object size and mirror occlusion.
  • The dataset described in the appendix was created using MNIST digits and textures from the Describable Textures Dataset.
Loading 2110.11405v3…