PaperlayerView exact revision on arXiv

Source-linked AI summary

GLIGEN: Open-Set Grounded Text-to-Image Generation

Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, Yong Jae Lee

arXiv:2301.07093v2cs.CVcs.AIcs.CLcs.GRcs.LG

TL;DR

Text-only generation limits precise spatial control and other conditioning, while separately trained conditional models do not preserve the knowledge of large pretrained diffusion systems. GLIGEN freezes pretrained weights and injects caption-plus-grounding information through new gated layers. It achieves open-world grounded generation and reports substantially stronger zero-shot layout performance than prior supervised baselines, while its scope includes specific grounding settings and evaluation comparisons.

  • Problem

    Text-only text-to-image models lack precise localization and other conditional controls, motivating methods that add grounding while retaining pretrained diffusion knowledge.

  • Method

    GLIGEN freezes the pretrained diffusion model and injects caption-plus-grounding inputs through new trainable gated layers.

  • Results

    GLIGEN achieves open-world grounded text-to-image generation, and its zero-shot layout-to-image performance significantly outperforms prior state-of-the-art methods.

  • Takeaways & Limitations

    The framework extends pretrained text-to-image models toward grounding with bounding boxes and can also accommodate keypoints, reference images, and spatially aligned conditions.

  • Takeaways & Limitations

    The main evaluation focuses primarily on bounding-box grounding, and average-precision comparisons are reported only against fine-tuned LDM because prior text-to-image methods do not accept box annotations.

Abstract

from arXiv · show

Large-scale text-to-image diffusion models have made amazing advances. However, the status quo is to use text input alone, which can impede controllability. In this work, we propose GLIGEN, Grounded-Language-to-Image Generation, a novel approach that builds upon and extends the functionality of existing pre-trained text-to-image diffusion models by enabling them to also be conditioned on grounding inputs. To preserve the vast concept knowledge of the pre-trained model, we freeze all of its weights and inject the grounding information into new trainable layers via a gated mechanism. Our model achieves open-world grounded text2img generation with caption and bounding box condition inputs, and the grounding ability generalizes well to novel spatial configurations and concepts. GLIGEN's zero-shot performance on COCO and LVIS outperforms that of existing supervised layout-to-image baselines by a large margin.

1. Introduction

GLIGEN addresses the limited controllability of text-only text-to-image models by adding grounding inputs while preserving pretrained concept knowledge. It freezes the original model, injects grounding information through gated trainable layers, and achieves open-world grounded generation with strong zero-shot layout performance.

  • Motivation: Existing text-to-image models lack precise localization and broader conditional control because they rely on natural language alone.Bounding boxes and keypoints can express spatial information more precisely than text, while existing multimodal methods rarely combine such inputs with controllable text-to-image generation.
  • Motivation: The paper asks whether pretrained diffusion models can acquire new conditional modalities while retaining their large-scale concept knowledge.This builds on the recognition paradigm of adapting foundation models rather than training each generation task independently.
  • Method: GLIGEN adds grounding inputs such as bounding boxes, reference images, and keypoints to pretrained text-to-image diffusion models while retaining caption conditioning.The central challenge is learning to inject grounding information without forgetting the pretrained model’s concept knowledge.
  • Method: The method freezes pretrained weights and adds trainable gated Transformer layers that gradually fuse grounding information during training and sampling.Scheduled sampling can use all layers early and only original layers later to balance grounding fidelity with image quality.
  • Method: GLIGEN uses the pretrained text encoder for grounded entity phrases and combines their encoded text features with location information.This shared text space supports grounding open-world vocabulary concepts associated with bounding boxes.
  • Results: GLIGEN achieves open-world grounded text-to-image generation, and its zero-shot layout-to-image performance significantly outperforms prior state-of-the-art methods.The reported contribution emphasizes novel localized concepts unobserved during training and the benefit of building on large pretrained generative models.

2. Related Work

Prior conditional generation methods explored text, layouts, masks, images, and other inputs, but were commonly trained independently and often supported only closed-set concepts. GLIGEN instead targets open-set grounded generation by building on pretrained text-to-image diffusion models.

  • Large-scale text-to-image models: Large-scale text-to-image systems include autoregressive and diffusion models, whose quality and concept coverage arise from stable objectives and large-scale training.Examples include DALL-E, Parti, DALL-E 2, Imagen, and latent diffusion models.
  • Large-scale text-to-image models: Make-A-Scene conditions generation on semantic maps but operates in a closed set of 158 categories, whereas GLIGEN targets open-world grounding.eDiff-I instead changes attention maps to make objects roughly follow semantic maps, while GLIGEN uses a box-based interface.
  • Image generation from layouts: Existing layout-to-image methods generate images from labeled bounding boxes but are closed-set, typically limited to concepts observed in datasets such as COCO’s 80 categories.These methods use architectures including VAEs, LSTMs, mask normalization, adversarial losses, and Transformers.
  • Image generation from layouts: GLIGEN is presented as the first open-set grounded image generation method, while concurrent ReCo obtains open-set abilities by fine-tuning the original Stable Diffusion weights.The comparison distinguishes GLIGEN’s preservation of pretrained weights from ReCo’s fine-tuning approach.
  • Other conditional generation: Other conditional image-generation methods use text, boxes, semantic masks, images, or diffusion cross-attention, but are usually trained independently from scratch.GLIGEN investigates how to build new conditional capabilities on existing pretrained models.

3. Preliminaries on Latent Diffusion Models

Latent diffusion models encode images into a latent space and learn iterative denoising conditioned on captions. Their pretrained denoising capability produces strong text-to-image results, but additional grounding inputs remain difficult to control.

  • Latent diffusion: Latent diffusion models reduce training cost by learning an image-to-latent mapping and training diffusion in the latent representation space.The first stage maps image x to latent z, and the second stage models generation over z.
  • Training objective: Starting from noisy latent z_T, the model iteratively produces less noisy samples conditioned on caption c at each timestep.The denoising objective trains f_θ to predict noise in latent representations.
  • Training objective: The LDM denoising objective minimizes the squared error between sampled noise and the model’s predicted noise over timesteps and latent representations.The timestep t is sampled uniformly, and f_θ is a timestep- and caption-conditioned denoising autoencoder.
  • Network architecture: A UNet with ResNet and Transformer blocks processes the noisy latent, timestep information, and caption features from a learned text encoder.Caption embeddings replace the caption input c in the denoising objective.
  • Motivation: Although internet-scale pretraining yields impressive caption-conditioned generation, instructing additional grounding inputs remains challenging.This limitation motivates extending pretrained latent diffusion models with grounding capabilities.

4. Open-set Grounded Image Generation

GLIGEN represents grounded instructions as captions plus entity-location pairs, encodes them into grounding tokens, and injects them through gated self-attention while keeping the pretrained model intact. The framework supports open-set concepts and extensions to images, keypoints, and spatially aligned maps.

  • Grounding instruction input: A grounded instruction combines a caption with grounded entities, each pairing semantic information e with a spatial configuration l.The spatial configuration may be a bounding box, keypoints, or an edge map.
  • Grounding instruction input: For bounding-box grounding, each entity token f_text(e) is fused with Fourier-embedded box coordinates through an MLP to form a grounding token.The box is represented by top-left and bottom-right coordinates, and grounding tokens are formed for all grounded entities.
  • Open-set grounding: GLIGEN uses the same pretrained text encoder for captions and grounded text entities, supporting open-world vocabulary instead of a fixed embedding dictionary.Closed-set dictionary lookup restricts models to observed entities, whereas shared text features enable text-described concepts.
  • Extensions: The framework generalizes grounding beyond boxes to image prompts, keypoints, and spatially aligned edge, depth, normal, or semantic maps.Spatially aligned maps can be encoded into grounding tokens and optionally downsampled for concatenation with the noisy latent at the UNet input.
  • Continual learning: GLIGEN freezes the original model weights and learns new modules so grounding information can be added without changing the pretrained components.This continual-learning design prioritizes retaining pretrained knowledge while expanding spatial grounding capabilities.
  • Gated self-attention: A new gated self-attention layer attends over visual and grounding tokens, selects visual outputs, and adds them as a gated residual between the original attention layers.The learnable gate is initialized at zero, enabling stable training while grounding information is gradually integrated.
  • Scheduled sampling: Scheduled sampling improves visual quality by using grounding early for coarse location and outline decisions, then relying more on original layers for later details.It also extends a human-keypoint-trained model to objects with human-like shapes, including monkeys and cartoon characters.

5. Experiments

Experiments evaluate GLIGEN for grounded text-to-image generation across closed-set, open-set, and additional grounding conditions. The results show strong image quality, grounding accuracy, open-world generalization, and benefits from scheduled sampling.

  • Experimental Setup: GLIGEN is evaluated on closed-set and open-set grounded text-to-image generation, primarily using pretrained LDM models.The experiments also examine extensions to other grounding modalities.
  • Experimental Setup: The model uses detection, caption-plus-detection, and GLIP-derived grounding instructions, with FID measuring image quality and YOLO score measuring box correspondence.Evaluation uses 30K generated images and compares predicted boxes with ground-truth boxes using average precision.
  • Closed-set Grounded Text2Img Generation: All three grounding instruction types produce comparable FID to the COCO-finetuned LDM baseline, while COCO2014D achieves the overall best performance.A model trained only with null captions has FID 8.2 when evaluated with real captions, indicating limited caption understanding.
  • Closed-set Grounded Text2Img Generation: GLIGEN achieves state-of-the-art performance for both image quality and grounding accuracy against layout-to-image baselines on COCO2017.The comparison uses the same detection annotations as the layout-to-image baselines.
  • Open-set Grounded Text2Img Generation: GLIGEN trained only on COCO localization annotations grounds novel concepts beyond COCO categories, including blue jay, croissant, and brown wooden table.The reported generalization is attributed to shared text spaces and gated self-attention repositioning visual features for grounded entities.
  • Open-set Grounded Text2Img Generation: On LVIS, COCO-trained GLIGEN outperforms a fully supervised LAMA baseline in zero-shot task transfer, while scaling training data improves zero-shot performance especially for rare concepts.GLIGEN runs on LVIS validation images without seeing LVIS labels; larger-data pretraining also supports further finetuning studies.
  • Other Grounding Conditions: Reference images, text-and-image inputs, and keypoints extend GLIGEN beyond bounding-box grounding, while keypoint training on humans generalizes to other humanoid objects.The keypoint model uses scheduled sampling to generate objects such as robots with human-like shapes.

6. Conclusion

GLIGEN expands pretrained text-to-image diffusion models with grounding ability while preserving their existing capabilities. Its open-world generalization and extensibility suggest a route toward broader controllable image synthesis.

  • Conclusion: GLIGEN expands pretrained text-to-image diffusion models with grounding ability and demonstrates open-world generalization using bounding boxes.The method is also described as extendable to keypoints, reference images, and spatially aligned conditions.
  • Conclusion: Scheduled sampling can improve visual quality or extend a model trained in one domain to others.The conclusion identifies this versatility as relevant to expanding pretrained models in applications.

A. Implementation and training details

GLIGEN represents grounding conditions as tokens or spatial inputs and injects them into a pretrained diffusion model through new gated layers. Its training setup supports boxes, images, keypoints, and spatially aligned maps.

  • Training setup: The implementation uses Stable Diffusion as an example and trains COCO experiments with 16 V100 GPUs, batch size 64, and 100k iterations.Scaling experiments use 400k iterations for LDM and 500K iterations with batch size 32 for Stable Diffusion; the learning rate is 5e-5.
  • Condition representations: Box grounding encodes each grounded phrase with the pretrained text encoder and combines it with Fourier-embedded bounding-box coordinates.The method uses the EOS text feature and Fourier embeddings with output dimension 64 for box coordinates.
  • Condition representations: Image grounding projects CLIP image features into the text feature space, while grounded entities can use up to 30 tokens and 60 tokens when both image and text represent an entity.The image encoder is ViT-L-14 for Stable Diffusion, and the maximum is 30 grounding tokens per representation.
  • Condition representations: Keypoint grounding adds learnable person embeddings so keypoints belonging to different people remain semantically associated.The person-token count is set to N = 10, with 17 keypoint annotations per person in COCO.
  • Condition representations: Spatially aligned conditions are resized to 256 × 256, encoded by ConvNeXt-Tiny into 64 grounding tokens, and optionally concatenated with the noisy latent input.The condition tensor is C × H × W, and the first U-Net convolution becomes trainable when the condition is added to the U-Net input.
  • Model integration: Grounding tokens are integrated through gated self-attention layers that project grounding tokens to the visual-token dimension before concatenation.The implementation inserts a projection layer into each Transformer block’s self-attention module.

B. Ablation Study

Ablations show that gated self-attention and natural caption handling are important for grounding quality, while Fourier coordinate embeddings strongly affect layout correspondence. Qualitative comparisons also report improved layout-to-image quality.

  • Attention ablation: Gated cross-attention reaches FID 5.8 but YOLO AP 16.6, versus YOLO AP 21.7 for gated self-attention.The comparison indicates that the ablated cross-attention design has similar image quality but worse grounding accuracy.
  • Caption ablation: Replacing the null caption with a noun list raises FID from 5.61 to 7.40 on COCO2014D.The authors attribute this degradation to the pretrained text encoder not encountering unnatural noun-list captions during LDM training.
  • Coordinate ablation: Fourier and MLP embeddings produce FID 5.82 and 5.80, but YOLO AP falls from 21.7 to 3.2 with MLP embeddings.The ablation separates image quality from layout correspondence: FID remains similar while grounding accuracy changes substantially.
  • Qualitative comparison: The model is reported to generate better-quality layout-to-image results, especially when using Stable Diffusion.The figure comparison uses baseline images copied from TwFA [71].

C.1. Text Grounded Inpainting

GLIGEN extends grounding beyond boxes to inpainting, keypoints, and reference images. The reported examples emphasize tighter spatial placement and additional caption-based control over generated people and scenes.

  • Text-grounded inpainting: Grounded inpainting places generated objects more tightly within the missing box than baselines.Baselines may fail to follow the box or ignore the caption when the missing category already appears elsewhere in the image.
  • Reference-image inpainting: Reference images can also ground the content inserted into missing regions.The inpainting implementation adds four channels for z0 and one for the inpainting mask to the first convolutional layer.
  • Keypoint conditioning: With human keypoints, the model produces higher-quality images than pix2pixHD and captions can specify scene or gender details.Both captioned and non-captioned model variants produce similar keypoint-correspondence results.

E. Additional quantitative results

Additional evaluations report stronger generation after LVIS finetuning and improved COCO layout correspondence from large-scale pretraining. These results reinforce the value of pretrained models for grounded generation.

  • LVIS evaluation: After finetuning on LVIS, GLIGEN produces more accurate generations and surpasses the supervised LAMA baseline by a large margin.The evaluation reports GLIP-score on the LVIS validation set.
  • COCO evaluation: Large-scale pretraining largely improves layout correspondence on the COCO2017 validation set after finetuning.The comparison reports image quality and layout correspondence against baselines.

F. Analysis on GLIGEN

GLIGEN’s gated self-attention establishes spatial correspondence between visual features and grounding tokens early in sampling, while later sampling prioritizes image refinement. Training eventually dampens the added layers’ contribution, but early UNet layers remain less interpretable.

  • Attention analysis: Visual features attend to grounding tokens with correct spatial correspondence from the first sampling step, even under Gaussian-noise input.This correspondence is observed in all eight attention heads of a middle UNet layer.
  • Attention analysis: The grounding-token correspondence fades during later sampling steps, aligning with scheduled sampling’s finding that rough layout is decided early.
  • Interpretability: Attention maps in the beginning UNet layers are less interpretable across all sampling steps, possibly because visual tokens lack positional embeddings.The authors hypothesize that later convolutional layers can leak position information through zero padding.
  • Training dynamics: The learned γ begins capturing spatial correspondence around 60–70k training iterations, near the plot’s peak.
  • Training dynamics: After learning spatial correspondence, the model dampens the new layers’ contribution while refining image quality and details with the original weights fixed.The authors present this as a hypothesis about the observed training dynamics.

G. More qualitative results

Qualitative results show that GLIGEN maintains competitive image quality while supporting bounding-box and multiple modality-based grounding conditions. Keypoints provide finer control than boxes but are less transferable across object categories.

  • Layout comparisons: GLIGEN has comparable image quality to layout2img baselines with LDM and more visual appeal and details with Stable Diffusion.
  • Keypoint grounding: Keypoint grounding combines text-to-image knowledge with keypoint annotations to control the keypoints of a specific person.The keypoint annotations come from COCO and are not linked with person identity.
  • Keypoint grounding: Keypoints transfer poorly from humans to categories such as cats or lamps, making them less generalizable than bounding boxes.Boxes specify coarse location and size and can therefore be shared across object categories, whereas keypoints represent object parts.
  • Bounding-box grounding: Bounding-box grounding produces controllable text-to-image results by grounding noun entities in captions.
  • Other modalities: GLIGEN demonstrates grounded generation with keypoints, HED maps, canny maps, depth maps, normal maps, and semantic maps.
Loading 2301.07093v2…