Source-linked AI summary
Rethinking Global Text Conditioning in Diffusion Transformers
Nikita Starodubcev, Daniil Pakhomov, Zongze Wu, Ilya Drobyshevskiy, Yuchen Liu, Zhonghao Wang, Yuqian Zhou, Zhe Lin, Dmitry Baranchuk
TL;DR
The paper asks whether pooled text conditioning is necessary or beneficial in diffusion transformers. It analyzes its conventional role and proposes using the pooled embedding as modulation guidance, finding improvements across models and tasks while identifying limits in text-to-image correspondence.
Problem
The paper investigates whether pooled text conditioning is necessary and whether it provides performance advantages over attention-based conditioning.
Method
The paper uses the pooled text embedding as modulation guidance and extends it to CLIP-free models with a small MLP added to the timestep embedding.
Results
The approach improves diffusion-model performance across text-to-image/video and image-editing tasks, including 22% better object counting and 18% better hands correction in human evaluation.
Takeaways & Limitations
Pooled text embeddings can be useful as guidance for controllable shifts toward improved aesthetic, complexity, and other visual properties, despite weak conventional influence.
Takeaways & Limitations
The method does not improve text-to-image correspondence and requires tuning a small number of additional hyperparameters.
Abstract
from arXiv · showhide
Diffusion transformers typically incorporate textual information via attention layers and a modulation mechanism using a pooled text embedding. Nevertheless, recent approaches discard modulation-based text conditioning and rely exclusively on attention. In this paper, we address whether modulation-based text conditioning is necessary and whether it can provide any performance advantage. Our analysis shows that, in its conventional usage, the pooled embedding contributes little to overall performance, suggesting that attention alone is generally sufficient for faithfully propagating prompt information. However, we reveal that the pooled embedding can provide significant gains when used from a different perspective-serving as guidance and enabling controllable shifts toward more desirable properties. This approach is training-free, simple to implement, incurs negligible runtime overhead, and can be applied to various diffusion models, bringing improvements across diverse tasks, including text-to-image/video generation and image editing.
1 INTRODUCTION
The paper revisits pooled text conditioning in diffusion models, finding that it contributes little conventionally but can improve generation when used as modulation guidance. The proposed approach is practical and improves state-of-the-art models across multiple tasks.
- The pooled embedding can act as a corrector that adjusts the diffusion trajectory toward better modes.
- The analysis finds that global text conditioning plays only a minor role relative to attention-based text conditioning.
- Modulation guidance can significantly improve performance and can be enhanced through dynamic strategies.
- The approach integrates pooled embeddings into fully attention-based models through modulation guidance.
- The method is simple to implement, incurs negligible overhead, and improves multi- and few-step diffusion models across text-to-image/video and image-editing tasks.
2 RELATED WORK
Related work improves diffusion models through classifier-free guidance, test-time optimization, and attention guidance. The paper differs by applying guidance in feature space through a small MLP rather than through attention.
- Classifier-free guidance research modifies scales, unconditional branches, and guidance dynamics to improve diffusion model quality.
- Test-time optimization methods commonly optimize handcrafted attention-based losses or the initial noise rather than the full denoising trajectory.
- Attention guidance methods extrapolate between positive and negative prompt attention outputs to control generation.
- The proposed approach uses guidance in feature space through a small MLP instead of applying guidance through attention.
3 MODULATION LAYERS
Diffusion transformers process image and text tokens with attention while using a global conditioning vector from the timestep and pooled text embedding to modulate layers. Modulation performs scaling and shifting, but its effect on image-generation quality is examined here.
- Transformer diffusion models process unified image and text token sequences through MLPs, normalization, and attention layers.
- The global conditioning vector combines the timestep with the prompt’s pooled embedding, while cross-attention incorporates text through the token sequence.
- The global conditioning vector is shared across the model and constructs a modulation space that influences modulation layers.
- Modulation layers use coefficients that scale and shift their inputs, and their role in improving image-generation quality remains unexplored.
- For long prompts, images without CLIP generally do not differ from the initial images according to DreamSim-based image differences.
4 ANALYSIS OF THE POOLED TEXT EMBEDDING ROLE
The paper investigates whether pooled text embeddings materially affect diffusion-model generation. Their conventional influence is weak or absent, varying by model and prompt length, which motivates using them differently.
- Recent diffusion models increasingly discard pooled text embeddings and rely on the timestep to produce global conditioning, without strict justification.
- In FLUX schnell, the pooled CLIP embedding is partially inactive, with negligible influence for long prompts but potentially impactful influence for short prompts.
- For long prompts in FLUX schnell, CLIP has little effect on image quality across CLIP Score, PickScore, and ImageReward.
- For longer prompts, the deviation between generations with and without CLIP becomes negligible and images resemble the initial generation.
- In HiDream-Fast, the pooled CLIP embedding has no effect for either short or long prompts.
- The pooled embedding also has no influence in COSMOS and in FLUX Kontext instruction-guided image editing.
5 MODULATION GUIDANCE
The pooled text embedding can guide diffusion models through modulation space, shifting generations toward desired local, global, and quality-related properties. Dynamic layer-wise guidance improves the quality–prompt correspondence trade-off while remaining training-free and computationally negligible.
- Modulation guidance reuses the pooled text embedding to steer diffusion trajectories toward modes with more desirable properties.The approach is training-free and amplifies the pooled embedding’s influence through guidance in modulation space.
- The guided modulation vector affects only shared modulation coefficients, incurring negligible computational overhead and working with CFG or distilled models.
- The pooled embedding can induce both local changes, such as hair length, and global changes, such as car style.These shifts are illustrated using positive and negative prompts representing contrasting attributes.
- Dynamic modulation guidance: Dynamic guidance adjusts the modulation scale across layers, rather than across diffusion time steps, to avoid excessively overweighting the prompt.The simplest strategy uses a step function that skips the first few layers.
- Dynamic modulation guidance: Dynamic guidance offers a better trade-off between aesthetic quality and prompt correspondence than constant guidance on 1K MJHQ prompts.Quality is measured with PickScore and text correspondence with CLIP score.
- Guidance shifts attention toward desired features, including hands and hand-related tokens, during hands correction.The effect is observed in both token-specific and averaged attention maps.
- For CLIP-free models, a small MLP can add pooled-embedding conditioning while the rest of the network remains frozen.The modified model behaves identically to the original when the pooled embedding is set to 0.
6 EXPERIMENTS
Experiments evaluate modulation guidance across text-to-image, video, and image-editing models and across general and specific generation objectives. The method improves several human and automatic measures, including specific-change preference, image quality, complexity, and video dynamics.
- The evaluation covers state-of-the-art text-to-image models, a CLIP-free model, video models, and the FLUX Kontext image-editing model.
- Human evaluations assess text relevance, aesthetics, complexity, and defects, while automatic metrics include ImageReward, HPSv3, GenEval, and VBench.
- Text-to-image generation: Aesthetics guidance improves both aesthetics and complexity, while complexity guidance mainly enhances complexity in text-to-image generation.ImageReward improves across models, and HPSv3 improves in most cases except SD3.5 Large with complexity guidance.
- Text-to-image generation: Introducing CLIP into COSMOS alone does not improve performance and can reduce complexity; gains appear when CLIP is combined with modulation guidance.
- Modulation guidance outperforms the original model and baseline approaches across the reported specific-change, comparison, and VBench evaluations.The cited table captions report improvements according to GenEval, human preference, and dynamic degree.
- Text-to-image generation: Human evaluation improves the original model by 22% in object counting and 18% in hands correction.These results use text relevance for object counting and defects for hands correction.
- Text-to-video generation: Both evaluated video models improve in dynamic degree, with particularly strong gains for CausVid.Incorporating CLIP alone provides no improvement in this experiment.
- Image editing: Modulation guidance mitigates FLUX Kontext’s difficulty with complex edits involving multiple objects.The method uses the final prompt as the positive direction and a blank prompt as the negative.
7 CONCLUSION
The paper concludes that the pooled text embedding has weak conventional influence but improves performance across tasks and models when used through modulation guidance. Dynamic guidance further outperforms constant guidance and provides practitioners with greater flexibility.
- The pooled text embedding has weak influence in conventional conditioning but improves performance across tasks and models through modulation guidance.
- Dynamic modulation guidance outperforms constant guidance, offering practitioners greater flexibility.The comparison is reported in the paper’s ablation studies.
A ADDITIONAL ANALYSIS FOR FLUX KONTEXT MODEL
FLUX Kontext editing results are unaffected by removing the pooled embedding, but the authors attribute this partly to CLIP’s mismatch with out-of-distribution editing instructions. Dynamic layer-level strategies are then investigated for targeted guidance.
- Removing the pooled embedding does not affect FLUX Kontext editing results, as confirmed visually and by CLIP score on SEED-Data.
- The authors suggest that out-of-distribution editing instructions for CLIP may contribute to weak editing strength in complex multi-object scenes.
- Using the final prompt as CLIP input with modulation guidance is proposed to address the editing mismatch.
- Attention maps are averaged over 1,000 local-feature prompts and 1,000 global-feature prompts to identify layer regions associated with each target aspect.
- Layer-level dynamic strategies are selected to match observed attention behavior, with strategies 3 and 4 improving hands correction and a step function outperforming constant scale for global changes.
C ABLATION STUDY
The ablation study shows that dynamic modulation guidance generally outperforms constant guidance, while its strength and starting layer determine the balance between enhancement, preservation, and artifacts. Modulation guidance also improves performance across CFG values and supports text-to-video generation.
- Dynamic modulation guidance: Dynamic guidance outperforms constant guidance for hands correction and object counting, with strategy 4 best for hands correction and strategies 1 and 4 tied for object counting.Strategy 1 is selected for object counting because of its simplicity.
- Dynamic modulation guidance: For aesthetics, strategy 1 performs best, while dynamic guidance balances quality enhancement and prompt correspondence better than constant guidance.Constant guidance can overweight the original prompt and cause divergence, whereas dynamic guidance permits higher guidance strengths without artifacts.
- Guidance strength and starting layer: Starting layers control the trade-off between preserving the original image and improving complexity; i = 18 and i = 28 preserve the image while enhancing fine-grained details.This result is obtained with fixed w = 3 under complexity guidance.
- Guidance strength and starting layer: Higher w improves the main object and background details under aesthetics guidance, but excessively large values such as w = 8 can introduce artifacts.The analysis fixes the starting layer at i = 5.
- Modulation guidance for different CFG: Modulation guidance improves performance across different CFG values, confirming that it is complementary to CFG.The evaluation uses the FLUX dev model with complexity guidance.
- Text-to-video generation: The same configuration used for aesthetics guidance in text-to-image generation makes videos more realistic and significantly improves their dynamic degree.The configuration is applied to text-to-video generation.
E BASELINES COMPARISONS FOR TEXT-TO-IMAGE GENERATION
The paper compares modulation guidance with attention-based, slider-based, and LLM-enhanced baselines across general and specific changes. It reports improvements without additional overhead, including stronger image editing and gains on complex cases that higher CFG alone does not resolve.
- General changes: Modulation guidance significantly improves aesthetics and complexity over Normalized Attention Guidance without incurring its additional attention-layer overhead.It can also be applied on top of LLM-enhanced prompts for further improvements.
- Specific changes: Modulation guidance outperforms LLM-enhanced prompts on hands correction and object counting, and improves over Concept Sliders without test-time optimization.For hands correction, it localizes attention without adversely affecting the rest of the image.
- Instruction-guided image editing: Modulation guidance consistently improves editing strength across all evaluated criteria in instruction-guided FLUX Kontext image editing.The evaluation measures Editing Strength and Reference Preservation using VLM scores.
- Instruction-guided image editing: Modulation guidance improves complex multi-object editing cases, whereas simply increasing CFG does not solve the problem.The comparison is illustrated for the FLUX Kontext model.
- Prompt-length evaluation: The study evaluates text-to-image performance separately for long and short prompts using automatic metrics on 1,000 prompts and human evaluation on 300 prompts.Table 11 reports aesthetics and complexity with and without modulation guidance.
H LIMITATIONS
The paper identifies two limitations: modulation guidance does not improve text-to-image correspondence, and achieving optimal performance requires tuning additional hyperparameters.
- Scope limitations: Modulation guidance does not address how accurately generated images reflect the input prompt.The limitation follows from its focus on aesthetic quality, complexity, and other visual attributes rather than semantic alignment.
- Configuration limitations: The method introduces additional hyperparameters that must be tuned for optimal performance.The tuning is described as relatively straightforward but adds a step compared with baselines requiring no such configuration.
- Evaluation setting: The evaluation includes side-by-side human preference comparisons across relevance, defects, aesthetics, and complexity.Professional assessors were trained and compensated, and the study used informed consent and anonymized data.