PaperlayerView exact revision on arXiv

Source-linked AI summary

SpaText: Spatio-Textual Representation for Controllable Image Generation

Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, Xi Yin

arXiv:2211.14305v2cs.CVcs.GRcs.LG

TL;DR

Fine-grained control over object shapes and layouts remains difficult for text-to-image models, and prior controls rely on fixed labels. SpaText introduces sparse open-vocabulary scene control using CLIP-based spatial text representations, extends diffusion guidance to multiple conditions, and reports state-of-the-art results across evaluations, while noting failures on crowded scenes and tiny segments.

  • Problem

    Text-to-image models struggle to control region shapes, object layouts, and spatial relations finely, while prior scene controls depend on fixed labels.

  • Method

    SpaText combines a global scene prompt with sparse segment maps annotated by free-form text, using a CLIP-based representation and multi-conditional classifier-free guidance.

  • Results

    SpaText achieves state-of-the-art free-form textual scene-control results, supported by automatic metrics, FID, and a user study in which human evaluators preferred the method.

  • Takeaways & Limitations

    Sparse open-vocabulary scene control gives users greater control over specified regions while leaving unspecified parts of the scene for the model to generate.

  • Takeaways & Limitations

    The model may miss or propagate segment characteristics when scenes contain more than a few segments and may ignore tiny segments below the effective resolution threshold.

Abstract

from arXiv · show

Recent text-to-image diffusion models are able to generate convincing results of unprecedented quality. However, it is nearly impossible to control the shapes of different regions/objects or their layout in a fine-grained fashion. Previous attempts to provide such controls were hindered by their reliance on a fixed set of labels. To this end, we present SpaText - a new method for text-to-image generation using open-vocabulary scene control. In addition to a global text prompt that describes the entire scene, the user provides a segmentation map where each region of interest is annotated by a free-form natural language description. Due to lack of large-scale datasets that have a detailed textual description for each region in the image, we choose to leverage the current large-scale text-to-image datasets and base our approach on a novel CLIP-based spatio-textual representation, and show its effectiveness on two state-of-the-art diffusion models: pixel-based and latent-based. In addition, we show how to extend the classifier-free guidance method in diffusion models to the multi-conditional case and present an alternative accelerated inference algorithm. Finally, we offer several automatic evaluation metrics and use them, in addition to FID scores and a user study, to evaluate our method and show that it achieves state-of-the-art results on image generation with free-form textual scene control.

1. Introduction

SpaText addresses the difficulty of controlling object layouts, shapes, and spatial relations in text-to-image generation by introducing sparse, open-vocabulary scene control. It combines a CLIP-based spatio-textual representation with multi-conditional diffusion guidance and evaluates the approach on pixel- and latent-based models.

  • Motivation: Text alone makes it practically impossible to specify precise object layouts, shapes, and spatial relations in generated images.The paper illustrates this limitation with a user trying to place a Labrador’s paw above a blue ball without touching it.
  • Motivation: Prior scene control uses dense segmentation maps with fixed labels, limiting objects outside the label set and requiring extensive pixel-level annotation.
  • SpaText: SpaText represents only user-specified regions with sparse spatial free-form text while leaving the rest of the scene unspecified.A global prompt describes the whole image, and each selected segment receives a local text description plus position and shape.
  • SpaText: Because detailed segment-level text datasets are unavailable, the method extracts local regions from existing image-text datasets and builds a CLIP-based representation.Training uses regions from a pre-trained panoptic segmentation model; inference embeds user descriptions with a CLIP text encoder and maps them into CLIP image-embedding space with a prior model.
  • Diffusion integration: SpaText is implemented with DALL·E 2 and Stable Diffusion, while classifier-free guidance is extended to multiple conditions and an accelerated variant trades controllability for inference time.
  • Evaluation: The authors evaluate SpaText with automatic metrics, FID, and a user study, reporting state-of-the-art results and preference from human evaluators.

2. Related Work

Related work covers text-to-image generation, scene-based generation, and localized text-driven image editing. SpaText is positioned as addressing scene control with general masks and free-form text, unlike prior fixed-label approaches.

  • Text-to-image generation: Text-to-image research progressed from low-resolution RNN and GAN systems in constrained domains to zero-shot open-domain transformer-based models.
  • Scene-based text-to-image generation: Prior scene-based generation studies did not provide general masks with free-form text control.
  • Scene-based text-to-image generation: Make-A-Scene improved controllability with dense segmentation maps but relied on a fixed set of labels.
  • Local text-driven image editing: Localized image editing methods modify existing images through inpainting or pretrained text-to-image models, offering a related route to scene-based generation.

3. Method

SpaText enables sparse, open-vocabulary scene control by combining a global text prompt with spatially localized free-form descriptions. It builds a CLIP-based spatio-textual representation from existing image-text data and integrates it into pixel- and latent-based diffusion models with multi-condition guidance.

  • Problem setting: The task combines a global scene prompt with a sparse spatio-textual matrix whose localized descriptions specify desired content at selected pixels.Unspecified pixels remain unconstrained by the user's local descriptions.
  • CLIP-based representation: Because segment-level textual datasets are unavailable, training extracts panoptic regions and embeds them with CLIPimg, while inference embeds local prompts with CLIPtxt and maps them through prior model P.The converted embeddings are placed at pixels indicated by the user's masks to construct ST.
  • CLIP-based representation: For each training image, K randomly selected disjoint segments are cropped, masked, resized, and embedded before their vectors are stacked in the corresponding segment shapes.The resulting representation ST has shape (H, W, dCLIP), with zero vectors outside selected segments.
  • Diffusion-model integration: The representation is incorporated into DALL·E 2-like pixel diffusion and Stable Diffusion by concatenating ST with image or latent features and fine-tuning the denoising components.The pixel model adapts decoder D, while the latent model uses the spatial correspondence of its fully convolutional autoencoder.
  • Multi-conditional guidance: Multi-conditional classifier-free guidance independently weights conditions during inference, enabling separate control of global text and local scene inputs.Figure 4 illustrates how changing condition scales trades local object attributes against global scene properties.

4. Experiments

Experiments compare SpaText with adapted baselines using automatic metrics, human ratings, qualitative examples, mask-sensitivity analysis, and ablations. SpaText (latent) generally achieves stronger text compliance and image quality, while mask inaccuracies can alter details without changing pose.

  • Quantitative & Qualitative Comparison: The evaluation compares SpaText variants with adapted No Token Left Behind and Make-A-Scene baselines using automatic metrics and human ratings.The automatic metrics assess image quality and compliance with global and local text; human ratings report majority preferences, with values above 50% favoring SpaText (latent).
  • Quantitative & Qualitative Comparison: SpaText (latent) outperforms the baselines in qualitative compliance with global and local prompts and in overall image quality.The comparison includes the two SpaText variants and baselines adapted to the task.
  • User Study: Human evaluators preferred SpaText (latent) over every baseline for overall quality, global text matching, and local text matching.The study used pairwise choices on Amazon Mechanical Turk across these three dimensions.
  • Mask Sensitivity: SpaText generates images that correspond to implicit masks imperfectly, as also reflected by local IOU scores.The authors suggest downsampling may remove fine-grained mask details, encouraging the model to fill gaps according to prompts.
  • Mask Sensitivity: The model is relatively insensitive to input-mask errors: local prompts can add ears or horns and change body type, while generated results retain the input pose.Examples include cat, dog, goat, and tortoise outputs from a general animal-shape mask.
  • Ablation Study: The baseline method outperforms ablations in FID, human visual quality, and human global text-match, but some ablations improve local IOU or local text-match.The binary representation improves local text-match but lowers local IOU; the CLIP-text-embedding variant has slightly better local IOU and human local text-match but worse FID and overall visual quality.

5. Limitations and Conclusions

SpaText addresses sparse scene control for text-to-image generation while acknowledging failures with crowded or tiny segments. The paper concludes that the method supports greater control over content generation.

  • Limitations: The model may miss segments or propagate their characteristics, especially when more than a few segments are specified.A blue bowl may become a beige vase matching the table’s appearance.
  • Limitations: The model struggles with tiny segments and may ignore their masks altogether.The authors suggest this may result from fine-tuning above a size threshold because low-resolution CLIP embeddings are meaningless.
  • Conclusion: The paper addresses text-to-image generation with sparse scene control.
  • Conclusion: The authors believe the method could support professional artists and novices by enabling greater control over content generation.

A. Additional Examples

The supplementary material provides additional model outputs, mask-insensitivity examples, multi-scale fine-grained control examples, and further limitations.

  • Additional Examples: Figures 8–12 provide additional results from the model.
  • Additional Examples: Figures 13–14 provide additional examples of the method’s mask insensitivity.
  • Additional Examples: Figures 15–16 show fine-grained control achievable with the multi-scale version of the method.
  • Additional Examples: Figure 17 provides additional limitations of the method.

B. Implementation Details

The implementation details describe the appendix organization and the training setup used for the two diffusion-based text-to-image models.

  • Implementation Details: The appendix covers diffusion-model, spatio-textual representation, baseline, and automatic-input implementation details.
  • Implementation Details: The approach uses DALL·E 2 and Stable Diffusion as its two state-of-the-art diffusion-based text-to-image models.
  • Implementation Details: The models were trained on a custom dataset of 35M image-text pairs following Make-A-Scene.

B.1.1 DALL·E 2 Implementation Details

The DALL·E 2 implementation combines prior, decoder, and super-resolution components, then fine-tunes the decoder for sparse scene control using a noise-prediction loss with a VLB term.

  • Model Components: DALL·E 2 consists of a prior, decoder, and super-resolution model composed as SR ◦D ◦P.The prior maps text representations to CLIP image embeddings, the decoder produces 64 × 64 images, and super-resolution upsamples them to 256 × 256.
  • Fine-Tuning: The decoder is fine-tuned for sparse scene control using the standard simple diffusion loss.
  • Fine-Tuning: The UNet predicts added noise from the noisy image, CLIP image embedding, spatio-textual representation, and timestep.
  • Fine-Tuning: The total training loss adds a variational lower bound term to the simple loss.The loss is written as L_hybrid = L_simple + λL_VLB.
  • Training Setup: λ was set to 0.001, with Adam optimization for 64,000 iterations at learning rate 6 × 10^-5.
  • Inference: During inference, DDIM uses 50 prior steps, 250 decoder steps, and 100 super-resolution steps.

B.1.2 Stable Diffusion Implementation Details

The Stable Diffusion implementation fine-tunes the latent denoising model to accept SpaText alongside text, using CLIP-based segment representations extracted from detected image regions.

  • B.1.2 Stable Diffusion Implementation Details: Stable Diffusion is adapted by adding the spatio-textual representation ST as an additional input to the latent denoising model.The autoencoder and CLIP text encoder remain frozen while the denoising model is fine-tuned.
  • B.1.2 Stable Diffusion Implementation Details: The training loss predicts noise from the noisy latent code, text prompt, spatio-textual representation, and diffusion timestep.
  • B.1.2 Stable Diffusion Implementation Details: Inference uses DDIM with 50 sampling steps.
  • B.1.2 Stable Diffusion Implementation Details: SpaText uses a pre-trained OpenAI ViT-L/14 CLIP model and a Detectron2 R101-FPN panoptic segmentation model.
  • B.1.2 Stable Diffusion Implementation Details: Training filters segments smaller than 5% of the image and randomly selects 1 ≤ K ≤3 segments for each spatio-textual representation.Small segments are filtered because their CLIP image embeddings are less meaningful for low-resolution images.
  • B.1.2 Stable Diffusion Implementation Details: Each input condition is dropped 10% of the time during training to enable multi-conditional classifier-free guidance.The model is trained fully unconditionally approximately 1% of the time.

B.4. Evaluation Dataset Details

The evaluation dataset is constructed from COCO validation images, combining global captions with randomly selected sparse segment prompts to create coherent scene-control inputs.

  • B.4. Evaluation Dataset Details: The evaluation procedure converts COCO validation captions and dense segmentation labels into global text and local prompts.Each segment label is represented as “a {label}”.
  • B.4. Evaluation Dataset Details: 30,000 input samples are generated by randomly selecting 1 ≤ K ≤3 segments to form sparse spatio-textual inputs.
  • B.4. Evaluation Dataset Details: Figure 18 provides random examples of the generated evaluation inputs.
  • B.4. Evaluation Dataset Details: The latent-based SpaText variant qualitatively outperforms baselines in global-text compliance, local-text compliance, and overall image quality.

B.5. User Study

The user study compares generated image pairs on visual quality and correspondence to global and local text, using thousands of ratings from Amazon Mechanical Turk evaluators.

  • B.5. User Study: The study evaluates overall image quality, global-text matching, and local-text matching to the spatio-textual mask.
  • B.5. User Study: Each comparison uses 512 COCO-derived inputs and five raters, yielding 2,560 ratings per task.
  • B.5. User Study: Raters choose the better image for the specified criterion, with equal outcomes handled through ties or equal preferences.
  • B.5. User Study: Table 3 compares parameter counts and single-image inference times across baselines and SpaText variants.

C. Additional Experiments

Additional experiments examine manual alternatives, mask robustness, multi-scale conditioning, random-label Make-A-Scene, and efficiency, while documenting practical limitations of scene control.

  • C. Additional Experiments: Multi-stage text-driven editing can place objects in desired locations but may produce less natural scene composition and poorer local-mask correspondence.Its iterative workflow is also more cumbersome for users.
  • C. Additional Experiments: Sketch-to-image methods require dense color sketches, whereas SpaText accepts local prompts and allows unspecified regions to be inferred.
  • C. Additional Experiments: Mask-insensitivity examples show local prompts changing body type or fine details while preserving the overall posture or shape.
  • C. Additional Experiments: Multi-scale inference trades off local and global conditioning, ranging from correct object attributes without context to global context without specified objects.
  • C. Additional Experiments: The random-label Make-A-Scene variant matches local prompts but performs worse than standard Make-A-Scene in automatic evaluation and the user study.
  • C. Additional Experiments: The method sometimes exhibits characteristic leakage between segments or from global text, and may ignore objects such as small balloons or the sun.
  • C. Additional Experiments: SpaText (latent) is significantly faster than the baselines and uses fewer parameters than Make-A-Scene and SpaText (pixel).
Loading 2211.14305v2…