PaperlayerView exact revision on arXiv

Source-linked AI summary

GRAN-TED: Generating Robust, Aligned, and Nuanced Text Embedding for Diffusion Models

Bozhou Li, Sihan Yang, Yushuo Guan, Ruichuan An, Xinlong Chen, Yang Shi, Pengfei Wan, Wentao Zhang, Yuanxing zhang

arXiv:2512.15560v3cs.CV

TL;DR

Text encoders shape the semantic fidelity of text-to-image and text-to-video diffusion, but adapting pretrained language representations for visual synthesis remains difficult. GRAN-TED introduces TED-6K for rapid encoder assessment and develops an encoder through visual-semantic fine-tuning followed by layer-wise weighting. The resulting encoder achieves state-of-the-art TED-6K performance and reported gains in both generation modalities.

  • Problem

    Text encoders shape semantic fidelity, but adapting pretrained language representations into precise, interpretable conditioning for visual synthesis remains nontrivial.

    Source: §1 · p. 2

  • Method

    GRAN-TED pairs the TED-6K text-only encoder benchmark with visual-semantic fine-tuning of Qwen3-VL-8B-Instruct and subsequent layer-wise feature weighting.

    Source: §1 · pp. 2–3

  • Results

    GRAN-TED achieves state-of-the-art TED-6K performance and demonstrable improvements in semantic accuracy and compositional coherence for both T2I and T2V generation.

    Source: §5 · p. 15

  • Takeaways & Limitations

    Under the reported setup, TED-6K evaluation takes about 4 minutes rather than about 50 hours for training a T2I diffusion model from scratch, enabling 750× faster iteration.

    Source: §1 · p. 3

Abstract

from arXiv · show

The text encoder is a critical component of text-to-image and text-to-video diffusion models, fundamentally determining the semantic fidelity of the generated content. However, its development has been hindered by two major challenges: the lack of an efficient evaluation framework that reliably predicts downstream generation performance, and the difficulty of effectively adapting pretrained language models for visual synthesis. To address these issues, we introduce GRAN-TED, a paradigm to Generate Robust, Aligned, and Nuanced Text Embeddings for Diffusion models. Our contribution is twofold. First, we propose TED-6K, a novel text-only benchmark that enables efficient and robust assessment of an encoder's representational quality without requiring costly end-to-end model training. We demonstrate that performance on TED-6K, standardized via a lightweight, unified adapter, strongly correlates with an encoder's effectiveness in downstream generation tasks. Notably, under our experimental setup, compared with training a diffusion model from scratch, evaluating with TED-6K is about \textbf{750$\times$ faster}. Second, guided by this validated framework, we develop a superior text encoder using a novel two-stage training paradigm. This process involves an initial fine-tuning stage on a Multimodal Large Language Model for better visual representation, followed by a layer-wise weighting method to extract more nuanced and potent text features. Our experiments show that the resulting GRAN-TED encoder not only achieves state-of-the-art performance on TED-6K but also leads to demonstrable performance gains in text-to-image and text-to-video generation. Our TED-6K dataset and evaluation code are available at the following link: https://anonymous.4open.science/r/GRAN-TED-4FCC/.

1 Introduction

Text encoders shape the semantic fidelity of diffusion-generated visuals, yet robustly evaluating and adapting their representations for visual synthesis remain challenging. GRAN-TED introduces TED-6K for fast encoder assessment and a two-stage approach to developing a text encoder.

1 Introduction

  • Text encoders represent semantic elements including attributes, spatial relationships, and temporal events, supporting compositional generalization and semantic fidelity between prompts and generated visuals.The paper says encoder quality fundamentally determines this semantic fidelity.Source: §1 · p. 2
  • Existing evaluation is difficult because NLP and retrieval measures do not directly match visual-generation needs, while end-to-end generative-model training is computationally prohibitive.Adapting pretrained LLM features into precise, unambiguous representations interpretable by diffusion models is also nontrivial.Source: §1 · p. 2
  • TED-6K evaluation takes about 4 minutes versus about 50 hours to train a T2I diffusion model from scratch, enabling 750× faster iteration in the reported setup.The benchmark’s representation-similarity assessment is standardized with a lightweight adapter and strongly correlates with downstream T2I and T2V effectiveness.Source: §1 · p. 3
  • The two-stage GRAN-TED approach fine-tunes Qwen3-VL-8B-Instruct for visual representation, then applies layer-wise weighting to extract nuanced text features.The authors report gains in T2I and T2V generation alongside robust TED-6K performance.Source: §1 · p. 3

2 Related Works

Prior work has developed richer text encoders and stronger ways to inject language features into diffusion systems. GRAN-TED instead focuses upstream on evaluating, selecting, and adapting encoders before costly generative-model training.

2.1 Text Encoder for Diffusion Models.

  • Recent work increasingly uses decoder-only LLMs and multimodal LLMs, following CLIP- and T5-family encoders, for instruction following, compositional understanding, and visually grounded semantics.Other methods strengthen language-feature integration into diffusion systems; GRAN-TED instead studies encoder evaluation, selection, and adaptation upstream of end-to-end training.Source: §2.1 · p. 3
  • Norm-Avg normalizes and averages hidden states across all layers, with cited work reporting stronger conditioning than a single layer or simple averaging.Many diffusion models instead use hidden states from a single deep layer.Source: §2.1 · p. 3
  • Post-training encoder refinement and text-encoder efficiency studies are complementary, while GRAN-TED does not target backbone compression, sampler acceleration, or smaller encoders.Its focus is representation prediction, text-only evaluation, and layer-wise feature fusion for DiT conditioning.Source: §2.1 · p. 4

2.2 Text-alignment Benchmarks for Text Encoders.

  • Output-alignment benchmarks score generated visuals rather than isolating the text encoder, so comparing encoders requires costly separate generative-model training.Examples include GenAIBench, VIEScore, GPT4-Eval, and video-oriented evaluations.Source: §2.2 · p. 4
  • TED-6K is positioned as an encoder diagnostic because pooled-vector retrieval and general LLM benchmarks do not match diffusion models’ use of full token sequences and static hidden representations.It measures preservation of fine-grained visually relevant semantics and whether representation scores predict downstream generation performance.Source: §2.2 · p. 4

3 Method

The method couples a text-only encoder benchmark with encoder development: TED-6K evaluates representations through a unified context aggregator, while GRAN-TED is built by specializing an MLLM and learning layer-wise features.

3.1 Overview

  • TED-6K screens encoder representations with a unified context aggregator, guiding selection of Qwen3-VL-8B-Instruct for targeted fine-tuning and layer fusion into GRAN-TED.The framework is designed to assess representational capabilities without end-to-end diffusion-model training.Source: §3.1 · pp. 4–5

3.2 Text-only Benchmark for Evaluation

  • TED-6K contains 6,641 instances pairing source captions with positive statements and semantically contradictory negatives, including hard negatives with high lexical similarity.The pipeline uses human verification to check positive matches and ensure negative statements are confusable with positives.Source: §3.2 · p. 6

3.3 Sentence-level Context Aggregator

  • A two-layer self-attention aggregator with a learnable context token is trained on paired captions and evaluates whether each source’s positive statement scores highest among candidates.Training pulls together context tokens from captions of the same image and pushes apart tokens from different images.Source: §3.3 · pp. 7–8

3.4 Specializing MLLMs as Text Encoders for Visual Generation

  • Fine-tuning Qwen3-VL-8B-Instruct on targeted VQA pairs from figures or videos and semantically rich captions produces GRAN-TED, specialized for diffusion-model text embeddings.The targeted questions cover visual-generation-relevant perspectives including object attributes, spatial relationships, and temporal order.Source: §3.4 · p. 8

3.5 Layer-wise Feature Weighting

  • Norm-Avg outperforms Avg, motivating layer-normalized hidden states with learnable softmax weights whose weighted sum learns task-relevant feature importance.The module is intended to select useful features from different hierarchical levels.Source: §3.5 · p. 9
  • Because layer weights may drift as diffusion training shifts from global structure to detail refinement, the method freezes them after an initial joint-training phase.The authors introduce this two-step strategy to provide a stable text representation for the remainder of training.Source: §3.5 · p. 9

4 Experiments

Experiments assess TED-6K against encoder comparisons and downstream generation, then evaluate GRAN-TED and its feature-weighting strategy. The reported results include gains in both generation modalities and analyses of benchmark behavior and stability.

4.1 TED-6K Results

  • GRAN-TED outperforms peer models on TED-6K and closes the gap with the 32B model; its gains span all semantic dimensions except Adjective and Adverb.It performs best on macro-level events such as Action, with room to improve on Spatial Relationship and OCR.Source: §4.1 · pp. 10–11, Table 1 · p. 10, Table 2 · p. 11
  • Decoder-only LLMs outperform encoder-only models on TED-6K, and the smaller 4B Qwen3-VL-Instruct beats the 5B UMT5-XXL.MLLMs also outperform their LLM backbones on the text-only benchmark, while instruction tuning has no consistent effect.Source: §4.1 · pp. 10–11
  • Norm-Avg generally outperforms single-layer features, but thinking models improve single-layer representations and degrade under Norm-Avg.TED-6K scores show a clearer positive correlation with model size under Norm-Avg than under single-layer extraction.Source: §4.1 · p. 11

4.2 Correlation with Generation Performance

  • The study validates its text-only benchmark by quantitatively examining score consistency with downstream generative performance, reporting correlations between benchmark scores and T2I and T2V generation performance.Table 3 covers T2I; all evaluated encoders are instruction models except UMT5-XXL, while Table 4 covers T2V.Source: §4.2 · p. 11, Table 3 · p. 12, Table 4 · p. 12

4.3 Ablation Study

  • Two-step training significantly improves over vanilla Norm-Avg with only one learnable parameter per Transformer layer; continuous learnable-weight training slightly degrades performance.Without the two-step strategy, layer weights remain unstable and non-monotonic, including after 200k steps.Source: §4.3 · p. 13, Table 5 · p. 13, Figure 4 · p. 13
  • +1.24 in GenAI and +0.007 in UnifiedReward for T2I, and +2.39 in GenAI and +0.136 in NewBench for T2V, versus Qwen3-VL-8B Norm-Avg.The corresponding TED-6K increase is +0.61.Source: §4.3 · p. 14

4.4 Further Analysis

  • Further analyses report that TED-6K distinguishes demanding downstream prompts, is stable across repeated runs, and favors a trained aggregator over mean pooling or QA-style assessment.Five aggregator runs vary by at most 0.02; shuffled caption-statement questions yield 27.05%–28.63% accuracy, near 25% random guessing.Source: §4.4 · pp. 14–15, Table 6 · p. 15

5 Conclusion

The paper introduces TED-6K for efficient text-encoder assessment and GRAN-TED, a two-stage encoder that performs strongly on the benchmark and improves text-to-image and text-to-video generation.

5 Conclusion

  • GRAN-TED achieves state-of-the-art performance on TED-6K and demonstrable improvements in semantic accuracy and compositional coherence for text-to-image and text-to-video generation.The encoder combines targeted visual-semantic fine-tuning with layer-wise weighting.Source: §5 · p. 15

A Aggregator Architecture

The context aggregator converts text-encoder representations into a sentence-level context embedding that mimics the text conditioning consumed by a DiT.

A Aggregator Architecture

  • The aggregator accepts either single-layer token embeddings or stacked multi-layer features, with layer fusion applied to multi-layer inputs.Layer outputs are normalized, then averaged or combined with learnable softmax weights; single-layer inputs skip fusion.Source: A Aggregator Architecture · p. 20
  • A learnable context token is prepended to the fused token sequence before the attention-based pooling module aggregates sentence-level semantics.The context token is shared across the batch.Source: A Aggregator Architecture · p. 20
  • A two-block self-attention Transformer stack lets valid text tokens and the context token attend to one another, while a padding mask excludes padded positions when available.Each block uses multi-head self-attention, a two-layer GELU MLP, and residual connections.Source: A Aggregator Architecture · p. 20
  • The aggregator applies LayerNorm and, when needed, a projection to the target dimension, then reads out the projected context token as the sentence-level embedding.This implements the mapping from token-level text embeddings and the learnable context token to the aggregated context embedding.Source: A Aggregator Architecture · pp. 20–21

B Theoretical Justification for the Two-Step Training Strategy

The theoretical argument links the changing denoising objective to drifting layer weights and non-stationary conditioning, motivating joint optimization followed by freezing those weights.

B Theoretical Justification for the Two-Step Training Strategy

  • The argument treats the two-step training strategy as a practical response to a drifting layer-weight optimum and the resulting non-stationary conditioning distribution.It proceeds from asymmetric roles for denoiser parameters and layer weights, through spectral bias, to the resolution of freezing the weights.Source: B Theoretical Justification for the… · p. 21

B.1 Premise 1: Asymmetric Roles in Joint Optimization

The appendix argues that layer weights adapt the conditioning distribution while the denoiser learns an evolving task; this creates a moving optimum and internal covariate shift, which the two-step strategy is designed to arrest.

B.1 Premise 1: Asymmetric Roles in Joint Optimization

  • The denoiser parameters θ optimize noise prediction for a given conditioning distribution, whereas layer weights ω shape that distribution for the current denoiser.The conditioning vector distribution is produced by the fusion module fω.Source: §B.1 · p. 21

B.2 Premise 2: Evolving Objective via Spectral Bias

  • Spectral bias makes diffusion training progress from learning low-frequency global structure to high-frequency details, so the conditioning information needed by the denoiser evolves.The passage associates coarse structure with large timesteps and low SNR, and local detail and texture with small timesteps and high SNR.Source: §B.2 · p. 22

B.3 Deduction: A Continuously Drifting Optimum for the Layer Weights

  • Because the denoiser's state and task evolve, the layer-weight configuration that minimizes loss for its current state also drifts, creating a moving target.The optimum is defined for a fixed denoiser state, and the argument says it changes as that state learns.Source: §B.3 · p. 22

B.4 Consequence: Non-Stationary Distribution and Instability

  • Continuously changing layer weights make the conditioning distribution non-stationary, so denoiser gradients rely on a stale distribution that can make training inefficient and potentially unstable.The text characterizes this mismatch as internal covariate shift within the training loop.Source: §B.4 · pp. 22–23

B.5 Resolution: The Two-Step Strategy

  • The two-step strategy jointly optimizes the denoiser and layer weights initially, then freezes the weights to stabilize conditioning for the remainder of training.The learned weights are described as a compromise across the initial dynamic objectives; stabilizing the distribution is said to facilitate more reliable and efficient convergence.Source: §B.5 · p. 23

C Image Examples

The section presents qualitative comparisons of text encoders across GenAIBench prompts using identical sampling settings. The examples include prompts specifying counts, colors, and spatial relations.

C Image Examples

  • The qualitative comparisons use identical sampling settings across text encoders on prompts from GenAIBench [25].The supplied examples identify prompts but do not state which encoder performs better.Source: C Image Examples · p. 23
  • “Two red balloons and three white clouds floating in the blue sky” specifies object counts and colors.Source: C Image Examples · p. 24
  • “Six birds soar above two towering trees” specifies counts for both birds and trees.Source: C Image Examples · p. 25
  • “Three ceramic cups sit to the right of a wooden fork” specifies both object counts and a spatial relation.Source: C Image Examples · p. 26

D.1 Experimental Setup

The setup evaluates text encoders with a unified context aggregator and trains diffusion models for text-to-image and text-to-video comparisons. Evaluation uses GenAI-Bench [25] with shared inference settings within each task.

D.1 Experimental Setup

  • The context aggregator is a two-layer attention network with hidden dimension 1024, trained for one epoch on 500k samples at learning rate 1e-5.Source: §D.1 · p. 28
  • The text-to-image models use Lumina-Image-2.0-2B [43], train from scratch at 256x256 for 144k steps, and jointly train layer weights ωi for the first 96k steps.The private dataset uses batch size 512; ωi are then frozen.Source: §D.1 · p. 28
  • The text-to-video setup fine-tunes Wan2.1-T2V-1.3B [52] for 20k steps, jointly training layer weights ωi for the first 12k steps before freezing them.Training interleaves image and video steps at a 7:3 ratio, with batch sizes 512 and 128, respectively.Source: §D.1 · p. 28
  • Both diffusion setups keep the backbone trainable and use learning rate 1e-4; inference uses 50 denoising steps and GenAI-Bench’s default negative prompt.CFG is 4.0 for text-to-image and 5.0 for text-to-video.Source: §D.1 · p. 28

D.2 Training Data Example

The appendix illustrates the contrastive data used to train the Context Aggregator and VQA data used to fine-tune GRAN-TED. It also lists prompts used to generate captions, benchmark statements, and contrastive-learning pairs.

D.2 Training Data Example

  • Aggregator training uses a source caption as anchor, a re-caption of the same visual content as positive, and captions of different visual content as negatives.The re-caption is semantically similar but textually distinct; examples cover image and video content.Source: Aggretator Training Data · p. 28, Figure 6 · p. 30
  • The GRAN-TED fine-tuning examples are VQA pairs for both image-based and video-based settings.The section identifies examples in Figure 5 but does not provide their contents in the text.Source: §D.2 · p. 28, GRAN-TED Training Data · p. 29, Figure 5 · p. 29

D.3 Prompt

Related papers

Loading 2512.15560v3…