PaperlayerView exact revision on arXiv

Source-linked AI summary

TIE: Training-free CLIP Embedding Guidance for Compositional Text-to-Image Alignment

Sujung Hong, Tae Eun Choi, Youngjun Jun, Chanyong Yoon, Seong Jae Hwang

arXiv:2610.32407v1cs.CV

TL;DR

Multi-object and multi-attribute prompts can produce missing concepts or incorrect attribute–object bindings. TIE analyzes CLIP embeddings and uses entropy-aware amplification and attention-based binding in a single training-free update. It improves compositional alignment across SDXL and SD3.5 without training or model modifications.

  • Problem

    Text-to-image models can fail to preserve requested objects and attributes in multi-concept prompts.

    Source: §1 · p. 1

  • Method

    TIE combines entropy-aware singular-value amplification with L0-attention-based binding in a single training-free text-embedding update.

    Source: §1 · p. 2, §6 · p. 8

  • Results

    TIE improves compositional alignment across SDXL and SD3.5 without training or model modifications.

    Source: §6 · p. 8

  • Takeaways & Limitations

    TIE addresses concept loss and confusion with minimal inference-time overhead while improving prompt-constraint satisfaction on multi-attribute and multi-object benchmarks.

    Source: §1 · p. 2

    The authors state that their measurements can be applied to longer and syntactically more complex prompts.

    Source: §3 · p. 3

Abstract

from arXiv · show

Text-to-image (T2I) diffusion models have advanced notably, but still fail to satisfy prompt conditions, resulting in attribute mismatches and missing objects. These errors normally fall into two categories: concept loss, when an object or attribute (i.e., concept) goes missing, and concept confusion, when an attribute is assigned to the wrong object or multiple objects blend together. Recent studies suggest that these challenges are caused by the limited capacity of CLIP text encoder to capture fine-grained semantic details. Although several methods have been proposed, concept loss and concept confusion persist in recent T2I models when handling multi-object, multi-attribute prompts. In this paper, we conduct an embedding-level analysis to develop a training-free, inference-time solution for concept loss and concept confusion across CLIP-based T2I models. We find that (1) concepts mentioned later exhibit higher embedding entropy, reflecting less concentrated representations that are more often associated with concept loss, and (2) the first CLIP attention layer captures the strength of binding between each object and its attribute. Guided by our findings, we introduce TIE, a method that improves semantic alignment through a single text-embedding update. TIE addresses concept loss via entropy-aware singular value amplification and resolves concept confusion through interpolation-extrapolation binding based on CLIP attention scores, all in a training-free manner. Extensive experiments demonstrate that TIE enhances semantic fidelity in multi-concept scenarios with minimal sampling overhead. The project page is available at https://hong-sujung.github.io/TIE/.

1. Introduction

Multi-object, multi-attribute prompts can lose requested concepts or bind attributes to the wrong objects. TIE uses CLIP embedding signals to address both failures through a training-free, single-step update.

1. Introduction

  • Concept loss omits a requested object or attribute, while concept confusion misaligns attributes with objects or merges objects.These failures are violations of prompt-specified constraints.Source: §1 · p. 1
  • CLIP text representations can miss fine-grained details, and existing approaches may target only specific failures, require supervision, or add sampling-time computation.The paper seeks a unified, efficient solution through analysis of CLIP embeddings.Source: §1 · p. 2
  • Later-mentioned concepts show higher token entropy, which reflects more diffuse information and serves as a proxy for concept-loss propensity.The paper also identifies first-layer CLIP attention scores as a signal of attribute–object binding strength.Source: §1 · p. 2
  • AEP amplifies tokens according to entropy to mitigate concept loss, while IEB strengthens intended and suppresses irrelevant attribute–object bindings to address concept confusion.Together they form TIE, which updates text embeddings in one step without training and applies to existing CLIP-based T2I models.Source: §1 · p. 2
  • TIE improves prompt-constraint satisfaction on multi-attribute and multi-object benchmarks with minimal inference-time overhead compared to baselines.The framework combines Adaptive Embedding Preservation (AEP) for concept loss and Interpolation-Extrapolation Binding (IEB) for concept confusion.Source: §1 · p. 2

2. Related Works

Prior work improves compositional generation through sampling-time interventions, text-embedding changes, and model post-training. The paper situates its embedding analysis among these approaches and notes that reward-based post-training operates at a different pipeline stage.

2. Related Works

  • Sampling-time methods modify attention, noise, or flow dynamics to improve compositional generation, but incur additional sampling-time computation.Examples include A&E, SynGen, EBAMA, InitNO, CONFORM, TACA, and FOCUS.Source: §2 · p. 2
  • Figure 2 presents positional bias and token entropy as signals related to concept loss, including comparisons across prompt positions and rendering states.Its panels compare two- and three-pair prompts and entropy for aligned, misaligned, present, mixed, or missing concepts.Source: Figure 2 · p. 3
  • Embedding-based methods modify CLIP representations, but these approaches tend to show weaker text-image alignment than cross-attention-based methods.Examples investigate token merging, geometric properties, and adaptive embedding modification.Source: §2 · pp. 2–3
  • Reward-based post-training optimizes the denoising network while keeping the text encoder fixed, whereas TIE modifies text conditioning without training.The paper describes the approaches as addressing different pipeline stages and potentially complementary.Source: §2 · p. 3

3. Analysis of Concept Loss and Confusion

The analysis examines token entropy as a signal of concept-loss susceptibility and L0 self-attention as a signal of attribute–object binding. It tests these signals in CLIP text-encoder representations and relates them to rendering failures.

3.1. Token Entropy and Concept Loss

  • Later-mentioned attribute–object pairs showed substantially more missing and misaligned attributes than earlier pairs in images where both objects appeared.The analysis manually categorized attributes in 220 generated images.Source: §3.1 · p. 3
  • In CLIP-G, entropy correlated with shared-direction alignment (|ρ| = 0.88); rotating embeddings toward that direction raised concept loss from 27.3% to 35.7%, while rotating away reduced it to 22.3%.Source: §3.1 · p. 4
  • Later-position tokens had higher entropy even when comparing nouns appearing in both positions, with a mean increase of 0.0033 (dz = 1.08; p = 1.9 × 10−9).Source: §3.1 · p. 4
  • Entropy was higher for misaligned, mixed, or missing concepts than correctly rendered ones, and increased as prompt-level rendering deteriorated.Concept loss frequency also rose across entropy quantiles; entropy provided a graded susceptibility signal, not a discrete prediction.Source: §3.1 · p. 4

3.2. Token Relationships in L0 Attention

  • Cosine similarity and L2 distance distributions substantially overlapped for related and unrelated attribute–object pairs, so final token embeddings alone did not clearly measure binding strength.Source: §3.2 · p. 4
  • At L0, target-attribute masking raised concept-loss rate by 58.5 points versus 1.5 for function-word masking despite 8.4× greater displacement; middle-layer masking raised it by 1.0 point.The findings support a role for L0 in establishing attribute–object binding.Source: §3.2 · pp. 4–5
  • After controlling for token distance, L0 attention still separated related from unrelated pairs, with AUCs of 0.648 for CLIP-L and 0.624 for CLIP-G after regression, and 0.634 and 0.658 after matching.The authors use L0 attention as a continuous, adaptive signal for IEB binding strength.Source: §3.2 · p. 5, Figure 3 · p. 5

4. Methods

TIE improves generation in multi-object, multi-attribute scenarios through two complementary components: AEP addresses concept loss, then IEB addresses concept confusion. AEP uses token entropy to guide embedding updates, while IEB uses first-layer attention scores to estimate binding strength.

4. Methods

  • TIE applies AEP to mitigate concept loss before IEB estimates binding strength from L0 attention scores to mitigate concept confusion.The two components are integrated to improve text-to-image generation in multi-object and multi-attribute scenarios.Source: §4 · p. 5

4.1. Adaptive Embedding Preservation

  • AEP uses entropy-aware SVD to amplify concept-token and EOT embeddings, strengthening representations according to their entropy to mitigate concept loss.Higher-entropy concepts receive stronger spectral concentration, while lower-entropy concepts are modified more conservatively.Source: §4.1 · pp. 5–7

4.2. Interpolation-Extrapolation Binding

  • IEB interpolates related attribute–object embeddings toward each other and extrapolates unrelated concepts apart, with update strength guided by L0 attention scores.Weakly bound related pairs receive stronger interpolation; unrelated pairs with stronger attention receive stronger repulsion.Source: §4.2 · p. 7

5. Experiment

The experiments compare TIE with baselines on SDXL and SD3.5 backbones using compositional prompt categories. Results report strong quantitative performance and qualitative improvements in semantic binding, concept preservation, and spatial consistency.

5. Experiment

  • Baselines include A&E, SynGen, Magnet, and ToMe on SDXL, FOCUS and TACA on SD3.5-Medium, and backbone-relative gains on SD3.5-Large.A&E and SynGen are adapted to SDXL base for fair comparison.Source: §5 · p. 8
  • The evaluation uses BLIP-VQA across Color, Shape, Texture, Complex, and Spatial categories, with 300 prompts and five fixed-seed images per category.Inference time is measured on an RTX A6000 GPU with batch size 1.Source: §5 · p. 8
  • Prompt parsing identifies attribute–object relations; L0 attention is averaged across heads, and token-level quantities are averaged for multi-token concepts.Source: §5 · p. 8

5.1. Results

  • TIE outperforms baselines across all SDXL categories, leads on Color, Shape, and Texture for SD3.5-Medium, and remains competitive on Spatial and Complex prompts.SD3.5-Large results also show consistent gains across model scales.Source: §5.1 · p. 8
  • Qualitative SD3.5 examples show that TIE alleviates concept loss and improves spatial consistency over baseline models.Additional qualitative examples across diverse models appear in the supplementary material.Source: §5.1 · p. 8

6. Conclusion

TIE targets concept loss and concept confusion in text-to-image generation using token entropy and L0 attention as indicators. The training-free framework combines AEP and IEB in a single embedding update, improving compositional alignment across SDXL and SD3.5 without training or model modifications.

6. Conclusion

  • TIE addresses concept loss and concept confusion using token entropy as a loss proxy and L0 attention as a binding-strength signal.Source: §6 · p. 8
  • The training-free framework combines entropy-aware singular value amplification in AEP with L0-attention-based binding in IEB through one embedding update.Source: §6 · p. 8
  • TIE improves compositional alignment across SDXL and SD3.5 without training or model modifications.Source: §6 · p. 8

Related papers

Loading 2610.32407v1…