PaperlayerView exact revision on arXiv

Source-linked AI summary

Text-to-Image Models Need Less from Text Encoders Than You Think

Nurit Spingarn, Noa Cohen, Tamar Rott Shaham, Tomer Michaeli

arXiv:2606.03715v1cs.CV

TL;DR

The paper asks which parts of rich text representations are actually needed to guide image generation. It constructs contextless embeddings that retain word meanings, multi-token word cohesion, and position, then compares them with full embeddings. BoPTW matches full-embedding results across challenging cases, suggesting that transformer-based diffusion models often rely mainly on word meanings and order.

  • Problem

    It remains unclear which aspects of rich text representations are actually utilized by image generation models, despite text encoders capturing complex linguistic structure.

  • Method

    The authors replace full text-encoder outputs with Bag-of-Tokens, Bag-of-Words, and Bag-of-Position-Tagged-Words embeddings, then evaluate them across pretrained diffusion transformers and diverse prompt benchmarks.

  • Results

    BoPTW achieves results on par with full text embeddings across challenging scenarios including counting, spatial relationships, and attribute binding.

  • Takeaways & Limitations

    TTI models often rely on individual word meanings and word order, while linguistic decoding is performed mostly by the image model.

  • Takeaways & Limitations

    The analysis focuses on pretrained transformer-based diffusion models; U-Net-based SD 2.1 and SDXL fail under BoPTW, with non-inferiority rates of 0.2% and 4%.

Abstract

from arXiv · show

Text-to-image models rely on text prompts as their primary interface to human intent. Prompts are encoded by a text encoder into embeddings that condition the image generation process. Beyond individual token meanings, text embeddings encode contextual information across the full prompt, such as compositionality and attribute binding. However, whether image models actually exploit this richer information remains underexplored. Here, we address the question: Which aspects of text representation are essential for image generation? We show that text-to-image diffusion transformer-based models commonly rely only on two relatively straightforward aspects of text representations: (i) the merging of adjacent tokens into a word representation, for words spanning multiple tokens, and (ii) word order, which is imprinted by the positional embedding of the text-encoder. To show this, we construct a new text embedding that encodes only individual word meanings and order but lacks any contextual information about the full prompt. We find that this bag of position-tagged words representation is sufficient to successfully guide image generation, achieving visual quality and text fidelity that are on par with full text embedding-guided generation. This demonstrates that, contrary to common belief, text-to-image models often do not use the rich information encoded in the text embedding beyond individual word meanings and word order. Instead, the decoding of complex linguistic structures is performed by the image model itself. Project webpage: https://nsping13.github.io/contextless-TTI/

1 Introduction

Text-to-image systems increasingly use expressive text encoders, but it remains unclear which parts of their rich representations image models actually exploit. This paper tests whether contextless representations retaining word-level information and order can guide generation as effectively as full embeddings.

  • Motivation: TTI models are commonly built on the assumption that richer text representations improve generation quality.Modern systems use increasingly powerful or multiple text encoders, including large pretrained language models.
  • Research question: The paper asks which aspects of text representations are essential for guiding image generation, rather than merely present in the encoding.Text encoders capture compositionality, attribute binding, and long-range dependencies, but their functional use by image models is uncertain.
  • Approach: The authors construct three contextless embeddings: Bag-of-Tokens, Bag-of-Words, and Bag-of-Position-Tagged-Words.These representations progressively add word cohesion and word-order information while excluding full-prompt context.
  • Evaluation: Across pretrained transformer-based diffusion models and diverse prompt benchmarks, the study compares simplified representations directly with full text embeddings.The evaluation covers SD 3, FLUX.1 Schnell, and FLUX.2 Klein-4B using prompts testing compositionality, counting, color fidelity, and attribute binding.
  • Findings: For complex prompts, BoPTW achieves results on par with full embeddings despite lacking compositional and other contextual interactions.The result includes counting, spatial relationships, and attribute binding.
  • Implication: The findings suggest that TTI models often use individual word meanings and word order rather than richer contextual information encoded by text embeddings.The authors argue that this reframes the respective roles of text encoders and image models.

2 Related Work

Prior work established the importance of text conditioning and studied both cross-attention and encoder behavior, while also documenting limited relational and compositional structure in CLIP-style representations.

  • Text conditioning: Imagen showed that large T5 and CLIP text embeddings substantially improve compositional reasoning and performance on complex prompts.This work helped establish the importance of large-scale text encoders in text-to-image generation.
  • Encoder interpretation: Most interpretability work has examined diffusion cross-attention, whereas fewer studies have directly analyzed the text encoder within TTI pipelines.The paper positions its approach as complementary to work that feeds intermediate text-encoder states into the diffusion model.
  • Linguistic structure: Prior studies report that CLIP-style encoders capture limited relational and compositional structure and can be insensitive to word order.These findings include bag-of-words-like behavior and invariance to relations and ordering.

3 Erasing contextual information from text embeddings

The paper removes contextual information from text embeddings while progressively retaining token identity, multi-token word cohesion, and absolute word position. These contextless variants replace standard encoder outputs and test what information image models require.

  • Standard representation: Standard text encoders jointly map token sequences into embeddings containing semantic content and contextual prompt structure.The proposed alternatives bypass this standard contextualization process.
  • Bag of Tokens: The BoT embedding represents each token independently by averaging its embeddings across unrelated sentences and positions.This removes information about the original prompt’s other tokens and token location.
  • Bag of Words: BoW preserves multi-token word cohesion by averaging sub-token embeddings only when they form the same complete word.This addresses ambiguities caused when different words share the same constituent tokens.
  • Prompt complexity: BoT and BoW suffice for relatively simple prompts, whereas BoPTW supports more complex prompts.The visual examples were generated with FLUX.1 Schnell.
  • Bag of Position-Tagged Words: BoPTW extends BoW by averaging word embeddings only across sentences where each word appears at the same absolute position.The positional restriction preserves word order while still removing surrounding prompt context.

4 Results and Discussion

Across transformer-based text-to-image models, contextless embeddings can preserve image-text alignment, with word-level tokenization and positional information providing the strongest results. However, older U-Net-based models remain highly dependent on richer text embeddings.

  • Experimental setup: The study evaluates SD 3, FLUX.1 Schnell, and FLUX.2 Klein-4B on DrawBench, GenEval, and MS-COCO prompts using VLM-based comparisons.The evaluated prompts cover compositionality, counting, color fidelity, and attribute binding.
  • Contextless embeddings: BoPTW embeddings combine word-level tokenization with positional information, consistently improving adherence for complex prompts over BoT and BoW embeddings.BoT lacks sufficient information for many complex prompts, while BoW resolves multi-token words and BoPTW adds absolute word position.
  • Quantitative comparisons: BoT embeddings exceed 40% non-inferiority in most settings, BoW exceeds 50% for most models and datasets, and BoPTW reaches 65% for most.The full-embedding baseline is typically 70%–90% across most models and datasets.
  • Category breakdown: BoPTW performs unevenly across categories, reaching 88%, 90%, and 100% on GenEval Single object but only 27%, 37%, and 24% on DrawBench Text.The values correspond to SD 3, FLUX.1, and FLUX.2, respectively.
  • DiT versus U-Net: SD 2.1 and SDXL largely fail with BoPTW, achieving non-inferiority rates of 0.2% and 4%, respectively, unlike the evaluated diffusion transformers.This suggests that newer DiT-based image models handle more linguistic interpretation than these U-Net-based models.

5 Conclusion

The paper concludes that transformer-based text-to-image models often need only word meanings and word order rather than full prompt context. It therefore reframes linguistic decoding as a capability handled largely by the image model.

  • Conclusion: BoPTW embeddings preserve text fidelity and visual quality while encoding individual word meanings and order without full prompt context.The conclusion proposes that word-level embeddings with explicit position tags could simplify the language-vision interface.
  • Future direction: The paper suggests broadening the representation from words to multi-word idioms as a possible way to improve results.This is presented as a future direction rather than an evaluated result.
  • Positional information: The positional knowledge analysis uses 230K MS-COCO tokens and finds that token embeddings encode ordinal position information.BoPTT averages token representations across sentences where each token appears at the same ordinal position.

A.2 Metrics

The supplementary metrics show that contextless embeddings maintain prompt adherence and image quality comparable to full embeddings across the reported evaluations.

  • Prompt adherence: CLIP-based prompt adherence scores for contextless embeddings are comparable to those obtained with full embeddings.The CLIP trends are consistent with the VLM evaluation.
  • Image quality: FID and KID scores indicate that image quality does not degrade across contextless embedding types.Both metrics are computed using the inception-v3-compat feature extractor, with lower scores corresponding to better results.

A.3 Additional results on the GenEval dataset

On GenEval, BoPTW-conditioned generations achieve comparable performance to full embeddings across categories and overall performance, with both successful and unsuccessful prompt examples shown.

  • BoPTW-conditioned generations achieve comparable GenEval performance to full embeddings across categories and overall performance.They occasionally surpass the full-embedding baseline in specific categories.
  • Figure S3 presents GenEval prompts where BoPTW embeddings were sufficient for the image model to adhere to the text.
  • Figure S4 presents GenEval prompts where BoPTW embeddings were insufficient for the image model to adhere to the text.
  • Table S4 reports GenEval scores for each category, with higher scores indicating better performance.

A.4 Additional results on the DrawBench dataset

DrawBench examples show that BoPTW embeddings are often sufficient for prompt adherence, while SD 2.1 and SDXL fail to effectively use them.

  • BoPTW embeddings are often sufficient to guide evaluated models toward outputs that faithfully adhere to DrawBench prompts.
  • Figure S8 shows successful DrawBench prompts where BoPTW embeddings were sufficient for prompt adherence.
  • Figure S9 shows failure cases where BoPTW embeddings lacked sufficient information and generated images semantically misaligned with the text.
  • For SD 2.1 and SDXL, BoPTW-guided images do not adhere to prompts and appear drawn from a similar distribution.These models appear to struggle to infer context and produce prompt-adherent results.
  • Figure S10 pairs BoPTW-generated images on the left with full-embedding generations on the right for SD 2.1 and SDXL.

B Experimental details

The experiments use five text-to-image models and multiple prompt datasets, with generation using constructed embeddings reported as similar to the original models.

  • All experiments were run on an NVIDIA RTX A6000.
  • Image generation with the constructed embeddings is similar to generation with the original text-to-image models.
  • The study uses five text-to-image models: SD 2.1, SD 3, SDXL, FLUX.1 Schnell, and FLUX.2 Klein.
  • Prompt datasets include MSCOCO 2014 and 2017, GenEval, DrawBench, and CC3M.
  • Gemma-3 and Claude-sonnet-4.5 are additionally used in the experimental setup.

B.2 Constructing contextless embeddings

Contextless embeddings are constructed from stored token vectors and generated unrelated contexts, while averaging more such contexts does not perceptibly change BoPTW generations.

  • Token embedding vectors are collected from CC3M and MSCOCO-2017 training data and stored for constructing contextless embeddings.
  • Additional sentences are generated for tokens appearing fewer than ten times and for words split into multiple sub-tokens.
  • Generated prompts preserve a target word at a specified zero-based position while varying the surrounding context.
  • The sentence-generation procedure requires exactly 10 strings in a valid Python list with no extra text.
  • Increasing the averaging pool of context-erasing sentences does not lead to any perceptible semantic change in BoPTW-guided generations.Figure S11 visualizes ten GenEval prompts generated with FLUX.1 Schnell across increasing averaging-set sizes.

B.3 VLM as a judge evaluation

The evaluation uses Gemma-3 as a VLM judge to compare images generated with full and contextless embeddings. A two-stage protocol first tests whether one image adheres significantly better to the prompt, then identifies the preferred image when a difference is detected.

  • Evaluation setup: Gemma-3 serves as a VLM judge for comparing full-embedding and contextless-embedding generations.Each comparison uses an image pair, the associated prompt, and randomized image order.
  • Two-stage protocol: The first stage asks whether one image adheres significantly better to the prompt than the other.The evaluator must answer “Yes” only when one image is clearly superior; otherwise it answers “No.”
  • Two-stage protocol: When the first-stage answer is “Yes,” the second stage asks the VLM to identify the superior image.The response is constrained to selecting either the first or second image.
  • Reported outcomes: The parsed outcomes are grouped into equal preference, full-embedding preference, or contextless-embedding preference.Equal preference corresponds to cases with no detected significant difference.
Loading 2606.03715v1…