PaperlayerView exact revision on arXiv

Source-linked AI summary

Free Lunch Alignment of Text-to-Image Diffusion Models without Preference Image Pairs

Jia Jun Cheng Xian, Muchen Li, Haotian Yang, Xin Tao, Pengfei Wan, Leonid Sigal, Renjie Liao

arXiv:2509.25771v1cs.CVcs.AI

TL;DR

Text-to-image alignment methods often depend on costly human preference annotations or learned reward models. This paper trains diffusion models to prefer matched prompts over LLM-generated mismatches, adapting DPO and KTO. The authors report consistent improvements over baselines without human preference annotations.

  • Problem

    T2I alignment methods rely on costly human preference annotations or reward modeling, while image preference pairs are expensive and may be noisy or inconsistent.

    Source: §1 · p. 2

  • Method

    TPO pairs dataset captions with LLM-generated mismatched prompts and adapts DPO and KTO into TDPO and TKTO.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    TDPO and TKTO consistently improve over their baselines and achieve state-of-the-art qualitative and quantitative performance without human preference annotations.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The framework is model-agnostic and can be integrated into RLHF-style pipelines while using text preferences instead of human-annotated image preferences.

    Source: §1 · p. 2, §5 · p. 9

    LLM-generated negatives may be less fluent or faithful, which can degrade alignment performance.

    Source: D LIMITATION · p. 21

Abstract

from arXiv · show

Recent advances in diffusion-based text-to-image (T2I) models have led to remarkable success in generating high-quality images from textual prompts. However, ensuring accurate alignment between the text and the generated image remains a significant challenge for state-of-the-art diffusion models. To address this, existing studies employ reinforcement learning with human feedback (RLHF) to align T2I outputs with human preferences. These methods, however, either rely directly on paired image preference data or require a learned reward function, both of which depend heavily on costly, high-quality human annotations and thus face scalability limitations. In this work, we introduce Text Preference Optimization (TPO), a framework that enables "free-lunch" alignment of T2I models, achieving alignment without the need for paired image preference data. TPO works by training the model to prefer matched prompts over mismatched prompts, which are constructed by perturbing original captions using a large language model. Our framework is general and compatible with existing preference-based algorithms. We extend both DPO and KTO to our setting, resulting in TDPO and TKTO. Quantitative and qualitative evaluations across multiple benchmarks show that our methods consistently outperform their original counterparts, delivering better human preference scores and improved text-to-image alignment. Our Open-source code is available at https://github.com/DSL-Lab/T2I-Free-Lunch-Alignment.

1 INTRODUCTION

Text-to-image models can generate high-quality images, but their outputs do not necessarily reflect human preferences or align precisely with prompts. This section motivates alignment without costly human-annotated image preferences and introduces text-pair optimization as an alternative.

1 INTRODUCTION

  • Pretraining on high-quality image-caption data does not necessarily align generated images with human preferences or the text.The paper therefore frames post-training alignment as a separate need.Source: §1 · p. 2
  • RLHF for text-to-image models requires a reliable reward model and substantial human-labeled data, making the pipeline complex and resource-intensive.The initial training dataset offers limited utility for this stage because preference-specific annotations must be collected separately.Source: §1 · p. 2
  • Diffusion-DPO and Diffusion-KTO simplify preference learning, but depend on costly, high-quality image-preference pairs that can be noisy and inconsistent.The section poses whether alignment can improve without annotating those pairs.Source: §1 · p. 2
  • TPO uses LLM-edited mismatched captions paired with original captions to align diffusion models without human preference-image annotations.The approach adapts DPO and KTO into TDPO and TKTO.Source: §1 · p. 2

2 RELATED WORKS

Prior work improves text-to-image alignment through changes to text processing and through preference optimization. This paper positions text preference optimization as a complementary, loss-agnostic approach to adapting preference learning for diffusion models.

Text-to-Image Models

  • Existing T2I alignment work modifies text encoders, embedding interactions, captioning, or inference, whereas this work focuses on preference optimization.The authors describe their approach as complementary to those directions.Source: §2 · p. 2

LLM Alignment

  • In LLM alignment, DPO represents preferences implicitly with a Bradley–Terry model and supports closed-form policy optimization.The paper adapts preference-optimization concepts developed for language alignment to the multimodal setting.Source: §2 · p. 3

Text-to-Image Preference Optimization

  • Diffusion-DPO and Diffusion-KTO adapt DPO and KTO to diffusion generators, while other work uses score matching to bridge language-model and T2I alignment.The passage names DSPO as the score-matching approach.Source: §2 · p. 3
  • The proposed framework is agnostic to the preference-alignment loss and can embed diffusion T2I objectives, illustrated with Diffusion-DPO and Diffusion-KTO.The paper presents these as two instantiations of its unified formulation.Source: §2 · p. 3

3 METHOD

The method constructs matched and mismatched text prompts for each image, then optimizes text preferences rather than image preferences. It instantiates this approach with DPO and KTO objectives for diffusion models.

3.1 TEXT PREFERENCE CONSTRUCTION WITH LLMS

  • Given a high-quality image-caption pair, the method keeps the original caption as matched and uses an LLM to create a visually distinct mismatched caption.This construction assumes the original dataset caption accurately describes its image; the authors motivate it as easier than collecting image-preference pairs.Source: §3 · pp. 3–4, §3.1 · p. 4
  • LLM prompt edits target content, object attributes, spatial arrangement or motion, and scene context or environment.The framework compares matched and mismatched input conditions and derives TDPO and TKTO from DPO and KTO.Source: §3.1 · pp. 4–5
  • The derivation assumes dataset captions are sampled independently of model parameters, so pθ(c) = pref(c), while pθ(x) ≠ pref(x).Under this assumption, optimizing Eq. (4) is equivalent to maximizing Eq. (1).Source: §3.1 · p. 5

3.2 TEXT PREFERENCE OPTIMIZATION FOR DIFFUSION MODEL

  • The method extends text preference optimization to diffusion T2I models using diffusion adaptations of TDPO and TKTO.The diffusion TDPO follows the Diffusion-DPO adaptation; the TKTO objective follows Diffusion-KTO and uses matched or mismatched prompt labels.Source: §3.2 · pp. 5–6

4 EXPERIMENTS

Across four datasets and five metrics, TDPO and TKTO generally outperform the baselines, with a few exceptions; qualitative examples show stronger capture of prompt details. The ablations also relate prompt-editing choices and implicit preference scores to alignment outcomes.

4.1 EXPERIMENTS SETUP

  • Experiments fine-tune only the SD v1.5 U-Net, using an SFT stage before TDPO, TKTO, and image-pair baselines, and evaluate with fixed sampling settings.Evaluation uses the same test prompts, with win rates averaged across five metrics; training uses HPSD, and tests include HPDv2, Pick-a-Pic v2, Parti-Prompts, and open-image-preferences-v1.Source: §4.1 · p. 6

4.2 RESULTS

  • TDPO and TKTO consistently surpass the baselines across four datasets and five metrics, except that Diffusion-KTO scores higher on a few metrics.The comparison reports average win rates against SD v1.5; the same constructed negative captions are used for the image-pair baselines.Source: §4.2 · p. 7, Table 1 · p. 7
  • Against methods trained on Pick-a-Pic preferences, TDPO consistently outperforms Diffusion-DPO, while TKTO trails Diffusion-KTO on HPS and IR scores.The authors report comparable performance without human annotations, using captions and winning images for training.Source: §4.2 · p. 7, Table 2 · p. 8
  • In the qualitative examples, TDPO and TKTO capture prompt details that baselines miss, including “twilight” and “misty,” a pursuing monster, and a forest setting.TKTO also renders two red flowers with directional lighting and deep shadows, unlike the baselines.Source: §4.2 · pp. 7–8, Figure 5 · p. 8

4.3 ABLATIONS AND DISCUSSIONS

  • Models with higher implicit preference scores also show higher human preference metrics, and TDPO and TKTO rank among the top performers on both measures.The score measures the model’s preference for an image with its matching caption over the same image with a mismatched caption.Source: §4.3 · pp. 8–9, Figure 6 · p. 9
  • Removing content-related prompt modifications significantly lowers CLIP score, while removing spatial modifications produces a trade-off and can improve TKTO.The authors suggest content changes affect sensitivity to prompt semantics, whereas spatial prompts can be ambiguous and destabilize learning.Source: §4.3 · p. 9, Table 3 · p. 9

5 CONCLUSION

The paper presents an alignment method using LLM-generated text-preference pairs to fine-tune text-to-image diffusion models without human preference annotations. Its TDPO and TKTO instantiations improve over baselines and are described as model-agnostic and compatible with RLHF-style pipelines.

5 CONCLUSION

  • LLM-generated text-preference pairs let the method fine-tune text-to-image diffusion models without human preference annotations.The conclusion characterizes this as “free lunch” alignment.Source: §5 · p. 9
  • TDPO and TKTO achieve consistent improvements over the baselines.Source: §5 · p. 9
  • The framework is model-agnostic and can be integrated into RLHF-style pipelines.Future work includes other preference-optimization algorithms and modalities, plus more diverse negative-sample generation.Source: §5 · p. 9

6 AUTHOR CONTRIBUTIONS

The author contributions distinguish project co-leadership and method development from algorithm design and evaluation. The appendix describes how implicit preference scores are computed and reports their relationship with human preference metrics.

6 AUTHOR CONTRIBUTIONS

  • Cheng Xian and Li co-led the project; Cheng Xian focused on algorithm design and evaluation, while Li developed the overarching method and framework.Other contributors provided technical feedback, supervision, research direction, revisions, and discussion of results.Source: §6 · p. 10

A.1 IMPLICIT PREFERENCE SCORE

  • Implicit preference scores use timestep t = 0.5, three samples per triplet, and averaged diffusion losses on HPSD evaluation triplets.The mismatched prompt is generated with a single modification to the original prompt.Source: §A.1 · p. 14
  • The authors report that higher mismatched-prompt diffusion loss relative to matched-prompt loss is associated with higher human preference scores.The appendix says this negative correlation aligns with the goal of improving text–image alignment.Source: §A.1 · p. 14

A.2 DATASET

The study evaluates on human-preference and prompt datasets, and uses HPSD for training and fine-tuning. Evaluation set sizes and dataset uses vary across the listed resources.

A.2 DATASET

  • HPDv2 and Pick-a-Pic v2 test sets provide 400 and 500 evaluation examples, respectively.HPDv2 contains human preference choices; Pick-a-Pic v2 is described as a human-feedback dataset.Source: §A.2 · p. 14
  • Parti-prompts contributes over 1,600 English prompts for evaluation, while open-image-preferences-v1 reserves its last 500 pairs for evaluation.The remaining open-image-preferences-v1 pairs form the training set.Source: §A.2 · p. 14
  • HPSD contains over 780,000 image-prompt pairs; the study uses all of it for first-stage SFT and its first 100,000 pairs for second-stage fine-tuning.Source: §A.2 · p. 14

A.3 TRAINING DETAILS

The appendix specifies training and checkpoint-selection settings, shows additional prompt-modification examples, and describes instructions for Gemini to generate modified prompts using the paper’s principles.

A.3 TRAINING DETAILS

  • Experiments use Stable Diffusion v1.5 on two A100 GPUs, with different batch sizes and learning rates for SFT and other training.SFT runs for 17,500 steps; checkpoints are selected using four metrics on images generated from 500 HPSD evaluation prompts.Source: §A.3 · p. 14

A.4 PROMPT MODIFICATION EXAMPLES

  • Figure 7 provides additional examples of prompt editing based on the paper’s modification principles.The subsection describes them as examples like those in Figure 2.Source: §A.4 · p. 14, Figure 7 · p. 15

A.5 MODIFICATION PROMPT INSTRUCTION FOR LLM

  • Gemini 2.0 Flash modifies original prompts using one or more strategies that follow the four modification principles.Figure 8 shows the instruction template and the available modification strategies.Source: §A.5 · p. 16, Figure 8 · p. 16

A.6 T2I ALIGNMENT’S STRONG CORRELATION WITH HUMAN PREFERENCE

The implicit preference score links text–image alignment with human preference, and qualitative examples show annotators favoring images that better match prompts. The training procedure clips negative-sample signals to stabilize optimization.

A.6 T2I ALIGNMENT’S STRONG CORRELATION WITH HUMAN PREFERENCE

  • The implicit preference score is strongly negatively correlated with all human-preference metrics, suggesting that human judgments are closely tied to text–image alignment.The score is defined as the diffusion-loss difference between positive and negative image–prompt pairs.Source: §A.6 · pp. 16–17
  • Annotators consistently prefer images that more faithfully depict prompt details, including Alice, two portals, floating, and hand-holding androids.The examples compare images generated from the same prompt.Source: §A.6 · p. 16, Figure 10 · p. 17

B.1 CLIPPING NEGATIVE GRADIENT

  • Direct TDPO and TKTO fine-tuning can degrade sample quality and become unstable; clipping the negative-sample squared L2 norm bounds extreme negative signals.For TKTO, the clamp is applied when the condition prompt does not match the image.Source: §B.1 · p. 18

B.2 HUMAN PREFERENCE METRIC PERFORMANCE

TDPO and TKTO score above baselines across human-preference metrics, while prompt-editing-budget results vary by metric. The section also presents qualitative comparisons and baseline setups.

B.4 MORE ABLATION STUDY: MODIFICATION BUDGET

  • Increasing the prompt-editing budget produces neither consistent improvement nor consistent degradation across PickScore, CLIP, HPS, and ImageReward.The ablation varies k from 1 to 3 distinct prompt-editing strategies for TDPO and TKTO.Source: §B.4 · p. 18

B.2 HUMAN PREFERENCE METRIC PERFORMANCE

  • TDPO and TKTO have higher average scores than baselines on almost all metrics and evaluation datasets.Figure 11’s radar chart compares TDPO and TKTO with Diffusion-DPO, Diffusion-KTO, SFT, and SD v1.5 across metrics.Source: §B.2 · p. 18, Figure 11 · p. 19

B.3 MORE QUALITATIVE COMPARISON

  • Figure 12 provides side-by-side generation comparisons, with prompts shown in the leftmost column.The passage introduces these images as additional qualitative comparisons.Source: §B.3 · p. 18, Figure 12 · p. 20

B.5 DIFF-DPO AND DIFF-KTO BASELINES

  • TDPO and TKTO outperform most baselines when image pairs or preference labels are absent, and outperform baselines across datasets for most metrics.The comparison includes constructed image pairs and a text-pair setup that uses modified prompts without generating negative images; Table 4 reports win rates across datasets and methods.Source: §B.5 · pp. 19–20

C THE USE OF LARGE LANGUAGE MODELS (LLMS)

D LIMITATION

The framework depends on prompt-editing quality and the generated negatives, while a fixed text encoder and single-LLM sampling constrain discrimination and negative diversity.

D LIMITATION

  • Prompt-editing strategies may miss subtle semantic distinctions or produce unnatural negative examples.The authors identify prompt-editing quality as a condition for the framework’s success.Source: D LIMITATION · pp. 20–21
  • Budget-constrained off-the-shelf LLMs can produce less fluent or faithful negatives, which can degrade alignment performance.Source: D LIMITATION · p. 21
  • Keeping the pretrained text encoder fixed may limit discrimination between closely related prompts, while using one LLM restricts hard-negative diversity.The authors point to adversarial, retrieval-based, or multi-model sampling as possible alternatives.Source: D LIMITATION · p. 21

E REPRODUCIBILITY STATEMENT

The paper outlines its method and evaluation setup and reports a comparison table of win rates across datasets and methods.

E REPRODUCIBILITY STATEMENT

  • The method, model architecture, training, and evaluation pipeline are outlined, with a consistent setup described for reproduction.Source: E REPRODUCIBILITY STATEMENT · p. 21
  • The paper says its code will be released and made available soon.Source: E REPRODUCIBILITY STATEMENT · p. 21
  • Table 4 compares win rates across datasets and methods, with the authors reporting that their methods outperform baselines for most metrics across datasets.The table identifies best and second-best results and distinguishes baseline methods from the authors’ methods.Source: Table 4 · p. 22
Loading 2509.25771v1…