PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Rewarding Large Vision-Language Models for Optimizing Prompts in Text-to-Image Generation

Hongji Yang, Yucheng Zhou, Wencheng Han, Jianbing Shen

arXiv:2505.16763v2cs.CV

TL;DR

Text-to-image prompt optimization must bridge the expertise needed to write effective prompts and the data demands of reward-model training. This paper uses one LVLM to rewrite prompts and reward generated images, iterating through self-rewarding training. It reports outperforming strong competitors on two datasets.

  • Problem

    Effective text-to-image prompting can require specialized expertise, while RL-based prompt optimization has required extensive image-reward-model training data and fixed reward models.

    Source: §1 · pp. 1–2

  • Method

    One LVLM rewrites user prompts and evaluates generated-image aesthetics and alignment as a reward, then iteratively trains through self-rewarding.

    Source: §1 · p. 2, §3.3.2 · p. 6

  • Results

    The method outperforms other methods on two datasets; on the larger DiffusionDB test set, it achieves a 0.3 improvement compared to Neuroprompt and Promptist.

    Source: §1 · p. 2, §4.3 · p. 8, §5 · p. 9

  • Takeaways & Limitations

    The framework is intended to help fill the gap between laymen and experts using text-to-image models.

    Source: §5 · p. 9

    The work could not use larger-parameter models because training and fine-tuning them requires computational resources that available hardware may constrain.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Text-to-image models are powerful for producing high-quality images based on given text prompts, but crafting these prompts often requires specialized vocabulary. To address this, existing methods train rewriting models with supervision from large amounts of manually annotated data and trained aesthetic assessment models. To alleviate the dependence on data scale for model training and the biases introduced by trained models, we propose a novel prompt optimization framework, designed to rephrase a simple user prompt into a sophisticated prompt to a text-to-image model. Specifically, we employ the large vision language models (LVLMs) as the solver to rewrite the user prompt, and concurrently, employ LVLMs as a reward model to score the aesthetics and alignment of the images generated by the optimized prompt. Instead of laborious human feedback, we exploit the prior knowledge of the LVLM to provide rewards, i.e., AI feedback. Simultaneously, the solver and the reward model are unified into one model and iterated in reinforcement learning to achieve self-improvement by giving a solution and judging itself. Results on two popular datasets demonstrate that our method outperforms other strong competitors.

1 Introduction

Text-to-image prompting can demand specialized expertise, while existing prompt-optimization approaches face costly annotation and reward-model data requirements. The paper proposes using an LVLM to rewrite prompts and provide rewards through iterative self-rewarding training, reporting state-of-the-art performance on two datasets.

1 Introduction

  • Text-to-image models can generate diverse high-quality images from user-provided prompts, but effective interaction requires specific expertise, including familiarity with specialized vocabularies such as “35mm” for camera parameters (Rombach et al., 2022; Saharia et al., 2022; Yu et al., 2022).Some prompt-rewriting methods train on large human-annotated datasets, which are prohibitively expensive; other efforts use calculated aesthetics and alignment metrics as rewards for reinforcement learning (Rosenman et al., 2023; Cao et al., 2023).Source: §1 · p. 1
  • Prior RL-based prompt optimization requires extensive training data for image reward models, and fixed reward models cannot improve alongside the proxy model.These constraints leave feedback static during training.Source: §1 · p. 2
  • An LVLM serves as both prompt-rewriting solver and image evaluator, with its aesthetic and alignment judgments used as rewards.The framework also uses the model to judge evaluator responses and construct preference pairs for training.Source: §1 · p. 2
  • Self-rewarding training iteratively improves prompt quality using a small amount of training data, addressing the need for larger, higher-quality RL datasets..Source: §1 · p. 2
  • The method achieves state-of-the-art performance against strong competitors on beautiful-prompt and DiffusionDB.The introduction reports comparisons on both datasets without giving metric values here.Source: §1 · p. 2

2 Background

The background frames prompt rewriting as a way to help text-to-image models realize their potential, and reviews how reward-generation choices shape reinforcement-learning approaches. It also situates the work among prompt-engineering, vision-language, and text-to-image research.

2.1 Prompt Rewriting

  • Prompt rewriting seeks to turn a user prompt into a more professional prompt that helps a text-to-image model produce more aesthetic and relevant images.Source: §2.1 · p. 2

2.2 Self-Rewarding Learning

  • RL fine-tuning optimizes responses using a reward function while KL regularization constrains the policy relative to a reference model.The section introduces preference modeling from paired responses and human preference labels before describing the LVLM-generated reward sequence.Source: §2.2 · pp. 2–3
  • Self-rewarding learning uses the same LVLM to generate rewards and iteratively train itself, rather than relying on a separately fixed reward source.The background contrasts predefined metrics, feedback models trained on annotated data, fixed external LVLMs, and a shared-weight rewriter and reward model.Source: §2.2 · p. 3, Figure 2 · p. 3

2.3 Related Work

  • Related prompt-optimization methods use defined aesthetic and alignment rewards, genetic keyword search, trained reward models with PPO, or constrained decoding (Hao et al., 2024; Pavlichenko and Ustalov, 2023; Cao et al., 2023; Rosenman et al., 2023).Source: §2.3 · p. 3
  • The background reviews vision-language models as multimodal models connecting language and images, alongside GAN-, VAE-, and diffusion-based text-to-image generation.It also describes prompt optimization and preference-training approaches including PPO and DPO (Rafailov et al., 2024).Source: §2.3 · pp. 3–4

3 Methodology

The methodology initializes an LVLM for prompt rewriting and image-preference assessment, then uses it as both solver and reward model in iterative DPO training. Its AI feedback scores generated images for aesthetics and instruction alignment.

3.1 Dataset Construction Pipeline

  • 104,487 prompt-rewrite pairs, generated-image evaluation data, and filtered response-judgment data provide training examples for rewriting and preference assessment.Evaluation examples are retained when VILA-40B scores agree with supplied facts; judgment examples are retained when standard answers score above irrelevant ones.Source: §3.1 · pp. 4–5

3.2 Initialization LVLM Capability of Rewriting and Assessing Preference

  • SFT initializes image-evaluation and response-judgment capabilities by fitting the model’s predictions to high-quality labeled responses.The objective approximates pθ(y|x) to the training-data distribution pdata(y|x).Source: §3.2 · p. 5

3.3 Training with Self-Rewarding

  • The same model generates solutions and rewards during each iteration, allowing its generative and reward capabilities to improve through an evolving feedback loop.The training process samples candidate rewrites, scores their generated images, constructs preference pairs, and trains with DPO over multiple iterations.Source: §3.3 · p. 5

3.3.1 Training on Rewriting

  • Prompt rewriting models the response as a conditional sequence generated autoregressively from the input prompt and previously generated tokens.The response is sampled from pθ(·|x), with generation continuing to the maximum length or an end token.Source: §3.3.1 · pp. 5–6

3.3.2 Training on Preference Assessing

  • Preference-assessing training teaches the model to judge a response to a prompt and image, while iterative optimization updates the reward ability alongside instruction-following.The RL objective uses r(x, y) and E(y, R) as reward functions for the prompt-response and response-judgment pair.Source: §3.3.2 · p. 6

3.4 RL from AI Feedback

  • For AI-feedback DPO, the solver generates candidate prompts, a fixed text-to-image model renders them, and the LVLM scores the images to form winner-loser prompt pairs.The highest- and lowest-scored images determine the corresponding winning and losing prompts.Source: §3.4 · p. 6

3.5 LVLM-as-a-judge for Aesthetics and Rewarding

  • The LVLM judging template evaluates aesthetics, PickScore preference, and instruction relevance, including object presence, count, and relationships.Aesthetic judgments consider composition, color, lighting, and visual appeal; PickScore asks whether the image represents the text in a way humans favor.Source: §3.5 · pp. 6–7

4 Experiments

Experiments compare the prompt optimizer with competing methods on BeautifulPrompt and DiffusionDB, then examine reward design, human preferences, data use, and generated-image examples. The reported results favor the proposed method across the two datasets and in the human study.

4.1 Settings

  • Experiments use Stable Diffusion v1.5 with 20 denoising steps and VILA-v1.5-8b; eight candidates are sampled per raw prompt for reward validation.Reward scores are averaged over three samples, and the highest- and lowest-scored candidates form preference pairs for two DPO iterations.Source: §4.1 · p. 7

4.3 Results

  • On BeautifulPrompt and DiffusionDB, the method outperforms other methods; it gains 0.1 in Aesthetic Score versus Beautiful-Prompt without degrading PickScore or CLIP, and reports a 0.3 performance gain over NeuroPrompt and Promptist on DiffusionDB (100k).The result is reported alongside evaluations on Beautiful-Prompt and DiffusionDB.Source: Table 1 · p. 7, Table 2 · p. 7, §4.3 · pp. 7–8

4.2 Comparative Methods

  • The comparison includes MagicPrompt, ChatGPT, Beautiful-Prompt, NeuroPrompts, Promptist, and human-engineered prompts, with simplified human prompts serving as the originals.ChatGPT is prompted to expand user prompts; the other named systems use open-source weights.Source: §4.2 · p. 7

4.4 Ablation Study

  • The ablations find that removing the Pick prompt yields a high aesthetic score but significantly lowers PickScore, while removing aesthetic or relevance prompts reduces aesthetic and CLIP scores; self-rewarding outperforms fixed-rewarding.The findings suggest a distinction between image aesthetics and human preferences.Source: §4.4 · p. 8, Table 3 · p. 8

4.5 Human Evaluation

  • In pairwise rankings by 20 volunteers, images from the optimized DPO model are selected most often, and DPO performs better than the SFT baseline at about a 55% win rate.Figure 4 compares LVLMDPO2 with User Prompt and LVLMSFT.Source: §4.5 · p. 8, Figure 4 · p. 8

4.6 Further Discussion

  • Most optimized prompts are 30–40 words long, aesthetic scores rise with prompt length, and the RL stage uses 20k prompts versus 40k for Beautiful-Prompt and 90k for Promptist.The 10k prompt set is a subset of the 20k set; evaluation accuracy on Pickapic-v1 increases after SFT and DPO training.Source: Figure 5 · p. 8, Table 4 · p. 8, Table 5 · p. 8

4.7 Quantitative Results

  • Examples show optimized prompts adding style and lighting details, correcting “Astronaut rides horse” to specify riding, and producing more aesthetic images than the bland images from original prompts.The discussion also describes expanding object details, such as interpreting “in 2456” as “futuristic.”Source: §4.7 · p. 9, Figure 6 · p. 9

5 Conclusion

The framework uses an LVLM for prompt rewriting and image aesthetics and alignment rewards, then iterates self-rewarding training to improve with limited data. On two datasets, it outperforms other methods.

5 Conclusion

  • On two datasets, the method outperforms other methods.The conclusion does not identify the datasets or provide metric values.Source: §5 · p. 9
  • The framework uses an LVLM to judge image aesthetics and alignment, providing AI feedback instead of human feedback.Source: §5 · p. 9
  • Self-rewarding training iteratively improves the LVLM based on limited data.The model achieves self-improvement through iterative training.Source: §5 · p. 9
  • The prompt optimization method is intended to bridge the gap between laypeople and experts using generative models.Source: §5 · p. 9

Limitations

The work is constrained by the computational resources needed to use larger-parameter models.

Limitations

  • Larger-parameter models could not be used because of substantial computational resource requirements.Source: Limitations · p. 10
  • Training and fine-tuning billion-parameter models such as VILA-40B demand substantial GPU memory and processing power.Source: Limitations · p. 10
  • Hardware constraints also make deploying large-parameter models in real-world applications challenging.Source: Limitations · p. 10

A Comparisons with the common tags

The discussion examines frequent words in optimized prompts and tests whether adding combinations of those words accounts for performance improvement.

A Comparisons with the common tags

B Further Discussion

The discussion examines candidate diversity in self-rewarding and the risk of bias or overfitting across iterations.

B Further Discussion

  • Candidate diversity affects self-rewarding performance, and more candidates can widen the winner-loser margin for RL fine-tuning.The discussion points to Table 9 for this relationship.Source: B Further Discussion · p. 12, Table 9 · p. 12
  • A third training iteration using an additional 5,000 data does not cause a significant performance drop.The results are reported in Table 10.Source: B Further Discussion · p. 12, Table 10 · p. 12
  • The self-reward model is at risk of bias or overfitting, with data quality, model capabilities, generative-model biases, and candidate count among possible contributors.Source: B Further Discussion · p. 12

C More Quantitative Results

This section presents visual comparisons of images generated with different prompts. It reports that optimized prompts produce more pleasing images and stronger alignment with the text.

C More Quantitative Results

  • The appendix presents visual comparisons of images generated with different prompts.Source: C More Quantitative Results · p. 12
  • The optimized prompt turns Minecraft’s flat, uninteresting view into a more aesthetically pleasing, detailed image.Source: C More Quantitative Results · p. 12
  • Examples such as “Riding a bike on mars” and “Galaxy cat” show optimized prompts adding text-aligned descriptions.The revisions add “a person riding a bike on mars” and “a space cat,” respectively.Source: C More Quantitative Results · p. 12

D Prompts for LVLM-as-a-judge

The appendix gives prompts for LVLM judges that assess generated images and evaluate responses about those images. The prompts specify separate scoring criteria for aesthetics, alignment, and response quality.

D Prompts for LVLM-as-a-judge

  • The judge compares images generated from original and candidate prompts to facilitate scoring on the same benchmarks.Source: D Prompts for LVLM-as-a-judge · p. 12
  • The image-evaluation prompts ask for separate, input-order-independent judgments of aesthetics and text-image alignment.The alignment prompt instructs the judge to consider image contents and then whether the described text appears in the image.Source: D Prompts for LVLM-as-a-judge · pp. 15–16
  • The response evaluator judges answers about the images for relevance, accuracy, objectivity, clarity, and completeness.It receives two responses describing the model’s prior image evaluations.Source: D Prompts for LVLM-as-a-judge · p. 17
  • Figure 8 identifies aesthetic, pick, and alignment scores as outputs of the LVLM-as-a-judge.The caption says scores are based on a rating system and the model is fine-tuned in advance with evaluation data to understand image aesthetics.Source: Figure 8 · p. 18
Loading 2505.16763v2…