Source-linked AI summary
ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment
Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, Gang Yu
TL;DR
CLIP-based diffusion models have difficulty aligning images with lengthy prompts containing multiple objects, attributes, and relationships. ELLA connects frozen LLM and diffusion components through a lightweight TSC and introduces DPG-Bench for dense-prompt evaluation. Experiments report superior semantic alignment and prompt following, including for community models and downstream tools.
Problem
CLIP-based diffusion models struggle to comprehend long, dense prompts involving multiple objects, detailed attributes, and complex relationships.
Method
ELLA uses a lightweight Timestep-Aware Semantic Connector to condition a frozen diffusion U-Net with timestep-dependent features from a pre-trained LLM, while DPG-Bench evaluates dense prompt following.
Results
ELLA demonstrates superior semantic alignment to existing state-of-the-art text-to-image models and enhances prompt following in community models and downstream tools.
Takeaways & Limitations
ELLA provides a lightweight way to add LLM-based dense-prompt understanding to existing diffusion models without training the U-Net or LLM.
Abstract
from arXiv · showhide
Diffusion models have demonstrated remarkable performance in the domain of text-to-image generation. However, most widely used models still employ CLIP as their text encoder, which constrains their ability to comprehend dense prompts, encompassing multiple objects, detailed attributes, complex relationships, long-text alignment, etc. In this paper, we introduce an Efficient Large Language Model Adapter, termed ELLA, which equips text-to-image diffusion models with powerful Large Language Models (LLM) to enhance text alignment without training of either U-Net or LLM. To seamlessly bridge two pre-trained models, we investigate a range of semantic alignment connector designs and propose a novel module, the Timestep-Aware Semantic Connector (TSC), which dynamically extracts timestep-dependent conditions from LLM. Our approach adapts semantic features at different stages of the denoising process, assisting diffusion models in interpreting lengthy and intricate prompts over sampling timesteps. Additionally, ELLA can be readily incorporated with community models and tools to improve their prompt-following capabilities. To assess text-to-image models in dense prompt following, we introduce Dense Prompt Graph Benchmark (DPG-Bench), a challenging benchmark consisting of 1K dense prompts. Extensive experiments demonstrate the superiority of ELLA in dense prompt following compared to state-of-the-art methods, particularly in multiple object compositions involving diverse attributes and relationships.
1 Introduction
ELLA addresses the difficulty CLIP-based diffusion models have with long, dense prompts by connecting frozen diffusion models and LLMs through a lightweight, timestep-aware adapter. The paper also introduces DPG-Bench and reports superior prompt alignment and improved community-model prompt following.
- CLIP-based diffusion models struggle with long prompts describing multiple objects, attributes, and relationships.
- DPG-Bench contains 1,065 lengthy, dense prompts covering multiple objects with diverse attributes and relationships.
- ELLA achieves superior semantic alignment against existing state-of-the-art models and improves prompt following in community models and downstream tools.
- ELLA equips existing CLIP-based diffusion models with powerful LLMs without training either the U-Net or LLM.
- TSC dynamically extracts timestep-dependent LLM conditions and adapts semantic features across denoising stages.
2 Related Work
Prior work improves compositional text-to-image generation through stronger text encoders, attention or latent manipulation, image-feedback optimization, and LLM-based prompt planning. These approaches can remain limited by base-model interpretability or CLIP's language understanding.
- Several diffusion models use LLM text encoders such as T5 alongside or instead of CLIP to improve text understanding.
- Attention-map and latent-manipulation methods impose spatial or semantic constraints but depend on base-model interpretability and provide coarse control.
- Reward-based approaches use image-understanding feedback to fine-tune text-to-image models, while remaining potentially constrained by CLIP text understanding.
- Other methods use LLMs to enhance embeddings or decompose prompts into regional descriptions for image-generation conditioning.
3 Method
ELLA bridges a pre-trained LLM and diffusion U-Net with a Timestep-Aware Semantic Connector that converts variable-length text features into fixed-length, timestep-conditioned semantic queries. Its training data is enriched with dense captions describing objects, attributes, and relationships.
- TSC receives arbitrary-length LLM text features and timestep embeddings, then outputs fixed-length semantic queries for conditioning noise prediction.
- Connector designs: An MLP offers a simple feature mapping, whereas a Resampler uses learned latent queries and cross-attention to handle arbitrary token lengths.
- Timestep-Aware Semantic Connector: Timestep conditioning lets the connector extract dynamic text features as diffusion models progress from main-scene prediction to detail refinement.
- Connector designs: The paper reports that TSC was selected after ablations of alternative connector designs.
- Dataset construction: CogVLM generates descriptive training captions specifying objects, surroundings, colors, textures, and spatial relationships.
4 Benchmark
DPG-Bench addresses the limited coverage of earlier benchmarks by evaluating dense prompts with longer descriptions, multiple objects, diverse attributes, and relationships. It combines richer prompt construction with hierarchical categories and automatic graph-based scoring.
- Earlier benchmarks average about 10–20 CLIP-tokenized prompt tokens and provide limited noun diversity for dense-prompt evaluation.
- Prompts are created from COCO, PartiPrompts, DSG-1k, and Object365 using generated scene details, attributes, and object relationships.
- The benchmark organizes content into 5 level-1 categories and 13 level-2 categories.
- Each prompt receives four generated images, mPLUG-large adjudication, question-level means, prompt scores, and an overall DPG-Bench mean score.
5 Experiments
Experiments evaluate ELLA on short compositional prompts, dense prompts, user preferences, downstream community models, and architectural ablations. Across these settings, ELLA improves prompt alignment while preserving comparable aesthetic quality and using relatively few trainable parameters.
- 5.1 Implementation Details: ELLA uses SDv1.5 and SDXL base models, trains on 34M image-text pairs, and additionally uses 100K high-quality 1024-resolution examples for ELLASDXL.
- 5.2 Performance Comparison and Analysis: On T2I-CompBench short compositional prompts, ELLASDv1.5 significantly surpasses SD v1.5 and is comparable to SDXL on some subsets.
- 5.2 Performance Comparison and Analysis: ELLA requires 0.06B trainable parameters for ELLASDv1.5 and 0.47B for ELLASDXL, with performance slightly trailing DALL-E 3.
- 5.2 Performance Comparison and Analysis: User preferences align with DPG-Bench outcomes, while ELLASDXL surpasses open-source models in text-image alignment with aesthetic quality comparable to SDXL.
- 5.2 Performance Comparison and Analysis: ELLA improves prompt following in six community models, including LoRA and ControlNet, while maintaining styles similar to the originals.
- 5.3 Ablation Study: LLaMA-2 13B outperforms TinyLlama 1.1B on short and complex prompts, while T5-XL performs better on short prompts but worse than LLaMA-2 13B on complex text.
- 5.3 Ablation Study: TSC attention emphasizes color and layout at higher noise levels, style-related words at lower noise levels, and keeps main entities consistently attended.
6 Conclusion and Limitation
The paper presents ELLA as a lightweight adapter that connects pretrained LLMs and diffusion models without training the LLM or U-Net, and introduces DPG-Bench for automatic dense-prompt evaluation. ELLA improves prompt following and can extend community models and downstream tools; future work considers image-text conditioning with MLLMs.
- ELLA equips diffusion models with pretrained LLMs through a lightweight Timestep-Aware Semantic Connector without training the LLM or U-Net.
- DPG-Bench provides automatic evaluation for long, dense prompts, while ELLA improves prompt following and integrates with community models and downstream tools.
- The paper identifies enhanced text-image alignment as a basis for future image-editing work and plans to investigate interleaved image-text conditioning with MLLMs.