PaperlayerView exact revision on arXiv

Source-linked AI summary

Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generation

Runhui Huang, Qihui Zhang, Zhe Liu, Yu Gao, Jie Wu, Hengshuang Zhao

arXiv:2607.11886v1cs.CV

TL;DR

Practical, reliable, preference-label-free reward models remain challenging for image-generation reinforcement learning. SpectraReward uses image-conditioned prompt likelihood from pretrained MLLMs, and experiments show consistent gains, with Self-SpectraReward outperforming AlphaGRPO by 6.3 points on TIIF-Bench and 2.1 on GenEval.

  • Problem

    Designing efficient, reliable, preference-label-free, off-the-shelf reward models for open-source image generators remains an open challenge.

  • Method

    SpectraReward scores generated images by the mean image-conditioned prompt-token log-likelihood from a frozen pretrained MLLM, while Self-SpectraReward uses a unified model’s understanding branch.

  • Results

    Self-SpectraReward outperforms AlphaGRPO by 6.3 points on TIIF-Bench and 2.1 points on GenEval, while matching or surpassing much larger MLLM reward models.

  • Takeaways & Limitations

    The results support reward-policy alignment as an important factor alongside reward-model scale for effective image-generation reinforcement learning.

  • Takeaways & Limitations

    Because it scores only original prompt likelihood, SpectraReward may under-emphasize implicit visual implications not directly expressed in the text.

Abstract

from arXiv · show

In this paper, we propose SpectraReward, a training-free reward function that turns pretrained MLLMs into off-the-shelf reward models for image-generation reinforcement learning. Instead of asking the MLLM to judge a generated image or answer decomposed verification questions, SpectraReward measures how well the original prompt can be recovered from the generated image through a single image-conditioned, teacher-forced forward pass. We use the average image-conditioned prompt log-likelihood as the reward, directly reusing the MLLM's pretrained image-text alignment ability without preference labels, reward-model fine-tuning. We further introduce Self-SpectraReward, a special case for unified multimodal models where the policy's own understanding branch serves as the reward model for its generation branch, forming a closed-loop self-improving framework without external reward models or external knowledge. Extensive experiments validate SpectraReward through a broad image-generation RL study covering two diffusion models, three RL algorithms, nine reward MLLM backbones from four MLLM families spanning 4B to 235B parameters, and five out-of-distribution text-to-image benchmarks. Results show that both SpectraReward and Self-SpectraReward significantly and consistently improve generation performance and outperform prior MLLM-derived reward training methods. Further analysis reveals that larger reward MLLMs are not always better, while Self-SpectraReward can match or surpass much larger external reward models, suggesting that reward-policy alignment is a key factor for effective image-generation RL. Project Page: https://huangrh99.github.io/SpectraReward/

1 Introduction

The introduction identifies the need for efficient, preference-label-free rewards for image-generation RL and proposes SpectraReward, which recovers prompts from generated images using pretrained MLLMs. It also presents Self-SpectraReward as a unified-model closed loop and reports broad validation across models, algorithms, backbones, and benchmarks.

  • SpectraReward: SpectraReward turns any pretrained MLLM into a training-free image-generation reward model without preference labels or additional training.It conditions the frozen MLLM on a generated image and uses one teacher-forced prompt pass to obtain token-level likelihoods.
  • SpectraReward: SpectraReward measures how well textual requirements can be read from a generated image through image-conditioned prompt-token likelihoods.The method reuses pretrained image-text alignment and avoids scalar judging and question decomposition.
  • Self-SpectraReward: Self-SpectraReward uses a unified multimodal model’s understanding branch to reward its generation branch without external models or knowledge.This forms a closed-loop self-improving framework within the same model.
  • Results: 6.3 and 2.1 gains over AlphaGRPO are reported on TIIF-Bench and GenEval, respectively.Self-SpectraReward also improves over SpectraReward with the best MLLM backbone by +1.2 on GenEval, +2.1 on GenEval2, and +2 on WISE.

2 Related Work

Prior image-generation reward models adapt RLHF using human preferences or prompt–image alignment data, but face bias against text-independent aesthetic details. Self-rewarding extends model-generated supervision from LLM alignment to unified multimodal models with integrated understanding and generation.

  • Reward Models for Image Generation: T2I reward models trained on human preferences or prompt–image alignment data provide alignment signals but are biased against text-independent aesthetic details.These models adapt RLHF to text-to-image systems, while their effectiveness is described as fundamentally bottlenecked by this bias.
  • Self-Rewarding in Unified Multimodal Models: Self-rewarding uses a model’s own evaluations for supervision and extends naturally to unified multimodal models that combine understanding and generation.Related efforts also align multimodal models through self-generated preference data and iterative self-evolution.

3 Method

SpectraReward turns a frozen pretrained MLLM into a training-free reward model by measuring image-conditioned prompt likelihood in one teacher-forced pass. Self-SpectraReward applies the same principle within unified multimodal models, while analyses show sensitivity to semantic errors and reliable group-wise ranking.

  • 3.1 SpectraReward: SpectraReward measures how well a generated image translates back into its prompt through a single image-conditioned, teacher-forced forward pass.It requires no preference labels, reward-model fine-tuning, or auxiliary pipeline.
  • 3.1 SpectraReward: The reward is the mean image-conditioned prompt log-likelihood, with higher values indicating that the image better supports the prompt’s semantic content.Token-wise likelihoods form a semantic spectrum that is aggregated into one scalar reward.
  • 3.2 Self-SpectraReward: Self-SpectraReward uses a unified model’s own image-conditioned understanding branch to score its generated images, creating a closed-loop self-improving framework.The design eliminates external reward models, additional large-scale MLLM serving resources, and pipeline complexity.
  • 3.2 Self-SpectraReward: Shared tokenization, vision encoding, and pretraining distribution align Self-SpectraReward’s understanding-based reward with the policy’s generation knowledge.Self-SpectraReward couples the generation branch’s text-to-image mapping with the understanding branch’s image-to-text measurement.
  • 3.3 Reward Signal Analysis: Token-level prompt likelihoods respond to targeted counting and object-identity errors, while their average remains a reliable scalar reward.Sequence-level rewards also remain higher for positive images, and SpectraReward ranks images with better prompt satisfaction above those with missing objects, wrong attributes, or incorrect spatial relations.

4 Experiments

Experiments show that SpectraReward and Self-SpectraReward consistently improve BAGEL across out-of-distribution benchmarks, while reward-policy alignment, backbone choice, and RL algorithm materially affect performance. Self-SpectraReward can match or exceed much larger external reward models, and AWM is selected for the main experiments.

  • Main Results: Both SpectraReward variants consistently improve the BAGEL baseline and AlphaGRPO across five out-of-distribution text-to-image benchmarks.The benchmarks are GenEval, TIIF-Bench, DPGBench, Geneval2, and WISE.
  • Main Results: At 512 resolution, SpectraReward improves BAGEL on TIIF-Bench overall short/long prompts by +10.0/+6.2, while Self-SpectraReward reaches 85.1/84.3.The reported short/long values correspond to the overall TIIF-Bench evaluation.
  • Main Results: At 1024-resolution inference, Self-SpectraReward reaches 89.8 on GenEval, 34.3 on GenEval2, and 0.76 on WISE, outperforming BAGEL and AlphaGRPO.WISE is the knowledge-grounded benchmark in this evaluation.
  • Reward MLLM Backbones: SpectraReward improves both SD3.5-M and BAGEL, with Gemma3, InternVL3.5, and Qwen3-VL all improving BAGEL on TIIF-Bench.The result supports the caption-likelihood reward’s architecture-agnostic behavior across diffusion models and reward MLLM families.
  • Reward MLLM Backbones: Self-SpectraReward achieves the best similar-scale result, matches the strongest 30B-class external reward, and outperforms Qwen3-VL-235B-A22B, while scale effects are non-monotonic.Within Qwen3-VL, 8B-to-30B scaling improves all reported metrics, but scaling to 235B causes a clear drop; Gemma3-12B-Pretrain also outperforms Gemma3-12B-Instruct across three benchmarks.
  • RL Algorithm: AWM achieves the best overall downstream performance among FlowGRPO, AWM, and DiffusionNFT, so the authors use AWM in the main experiments.The comparison is conducted in the Self-SpectraReward setting.
  • Reward Function: The reward function substantially affects RL performance: SpectraReward is compared with scalar scoring and VQA-Score as alternative MLLM-based image-text alignment rewards.Scalar scoring rates alignment from 1 to 5, whereas VQA-Score uses the probability of the “yes” token.

5 Conclusion · Appendix

The conclusion presents SpectraReward as a training-free, image-conditioned prompt-likelihood reward for using pretrained MLLMs in image-generation reinforcement learning, alongside Self-SpectraReward for unified multimodal models. The Appendix discusses limitations, implementation details, and additional experimental results.

  • 5 Conclusion: SpectraReward turns pretrained MLLMs into off-the-shelf reward models by measuring image-conditioned prompt likelihood without massive preference labels or reward-model fine-tuning.It aggregates token-level semantic spectra induced by an MLLM into a simple reward.
  • 5 Conclusion: SpectraReward is introduced as a training-free reward function for image-generation reinforcement learning.The method reuses pretrained MLLMs rather than requiring reward-model training.
  • 5 Conclusion: Self-SpectraReward is introduced for unified multimodal models as a related extension of the proposed reward framework.The supplied conclusion passage identifies Self-SpectraReward but does not provide further mechanism details.
  • Appendix: The Appendix is organized around limitations and future work, implementation details, and additional experimental results.The listed topics include prompt-likelihood limitations, reward-prompt formatting, EOS masking, VAE features, EOS-token removal, and VAE-feature effects.
  • Appendix: The limitations discussion covers dependence on MLLM visual reasoning, implicit physical and commonsense implications, and complementary reward signals.These issues are presented as limitations of prompt-likelihood rewards.
  • Appendix: The implementation details include reward prompt format, EOS masking, and VAE-feature use in Self-SpectraReward on BAGEL.The passage specifically identifies VAE features in Self-SpectraReward on BAGEL as an implementation topic.
  • Appendix: The additional experimental results examine EOS-token removal and the effect of VAE features in reward calculation.These experiments are listed separately from the implementation details.

A Limitations and future work

SpectraReward’s quality is limited by the chosen MLLM’s visual understanding and reasoning, while prompt-likelihood scoring primarily measures explicit text-image semantic alignment and may miss implicit visual implications.

  • Limitations: SpectraReward’s reward quality is bounded by the visual understanding and reasoning ability of its chosen pretrained MLLM backbone.The method relies on image-conditioned prompt likelihoods from the reward backbone.
  • Limitations: Because likelihood is computed only over the input prompt, SpectraReward mainly captures explicit semantic alignment between generated images and prompts.This scoring design focuses on semantics directly expressed in the text.
  • Limitations: SpectraReward may under-emphasize implicit visual implications that are not directly expressed in the prompt, such as associations suggested by “hot coffee.”The passage gives “hot coffee” as an example of a prompt whose implicit visual implications may be missed.

B Additional Implementation Details

SpectraReward scores generated images by teacher-forced recovery of the original prompt without a general prefix, while excluding [EOS] from the computation; on BAGEL, Self-SpectraReward includes VAE features by default.

  • SpectraReward computation: SpectraReward feeds the generated image as visual condition and teacher-forces the original prompt, omitting a general descriptive prefix by default.A prefix may change absolute likelihoods but does not substantially affect within-prompt relative rankings used by group-relative RL; an exception is noted in the passage.
  • SpectraReward computation: Excluding [EOS] removes sequence-termination effects that can disproportionately influence short prompts’ averaged token likelihoods.The ablation reports improved GenEval with comparable TIIF-Bench performance.
  • Self-SpectraReward on BAGEL: For BAGEL Self-SpectraReward, the model’s understanding branch scores generated images using semantic visual and VAE features, with VAE features included by default.The default VAE input gives better average performance in the reported ablation.

C Additional Experimental Results

Additional experiments show that EOS-token handling matters because EOS can distort averaged prompt likelihoods without conveying image-text alignment. On BAGEL, incorporating VAE features into Self-SpectraReward improves generation evaluation results.

  • EOS token removal: EOS tokens often have much lower likelihood than regular prompt tokens, disproportionately shifting averaged SpectraReward scores, especially for short prompts.The effect is strongest when reward averages token log-likelihoods.
  • VAE features: Including VAE features in BAGEL’s Self-SpectraReward computation improves GenEval and TIIF-Short performance.BAGEL combines ViT semantic-encoder features with VAE generation-encoder features in its understanding branch.
  • EOS token removal: SpectraReward masks EOS because it only marks sequence termination and carries no semantic information about image-text alignment.Masking prevents this non-semantic token from affecting reward computation.

D Detailed Benchmark Results

Detailed results across GenEval, TIIF-Bench, DPG-Bench, and WISE show that SpectraReward and Self-SpectraReward consistently improve the BAGEL baseline across diverse text-to-image requirements. Gains are especially evident for compositional, instruction-following, prompt-following, and world-knowledge evaluations, with method-specific strengths varying by benchmark and resolution.

  • Overall detailed results: Across GenEval, TIIF-Bench, DPG-Bench, and WISE, both methods improve BAGEL across diverse text-to-image requirements.The benchmarks cover compositional generation, instruction fidelity, decomposed prompt following, and world knowledge or semantic reasoning.
  • GenEval: On GenEval, both methods improve BAGEL and AlphaGRPO at 512 and 1024 resolutions, with the largest gains in counting, spatial position, and color-attribute binding.These categories require correct attribute binding to objects, not merely object presence.
  • TIIF-Bench: On TIIF-Bench, both methods substantially improve BAGEL and AlphaGRPO on overall short- and long-prompt scores across basic, advanced, and designer-oriented instruction following.Improvements include basic attribute and relation following and advanced combinations of attributes and relations.
  • DPG-Bench: On DPG-Bench, SpectraReward achieves the best 512-resolution overall score, while Self-SpectraReward leads the attribute score and achieves the best 1024-resolution overall score.SpectraReward improves the global, relation, and other categories, whereas Self-SpectraReward has the strongest attribute score.
  • WISE: With self-CoT on WISE, both methods improve BAGEL and AlphaGRPO, and Self-SpectraReward reaches the best overall score while improving space, biology, physics, and chemistry domains.WISE emphasizes world knowledge and semantic reasoning in text-to-image generation.

E More Visualizations

Additional visualizations show that SpectraReward ranks images by fine-grained prompt satisfaction, while Self-SpectraReward improves spatial, counting, attribute, and object-relation grounding over BAGEL. These qualitative examples support the reported instruction-following gains beyond global image quality.

  • Reward ranking: Higher SpectraReward scores correspond to better satisfaction of object, attribute, and spatial constraints within rollout groups.This supports image-conditioned prompt likelihood as a group-wise ranking signal.
  • Qualitative comparisons: The BAGEL comparisons highlight improvements in spatial relations, counting, attribute binding, relative size, and object co-occurrence, including long prompts.These cases indicate finer-grained prompt grounding rather than only improved global image quality.
  • Qualitative comparisons: Self-SpectraReward more often preserves requested object relations and recognizable visual content than the BAGEL baseline.Examples include placing a toy car on modeling clay and separating a pen from a fabric blanket.

F Broader Impacts

SpectraReward reduces the cost and resource requirements of constructing image-generation RL rewards by reusing frozen pretrained MLLMs without preference annotation or reward-model fine-tuning. However, easier optimization may amplify misuse and inherit biases or visual misjudgments from the reward backbone.

  • Benefits: SpectraReward makes image-generation RL more accessible by reusing frozen pretrained MLLMs without preference annotation or reward-model fine-tuning.This can support research on prompt following, compositional generation, and unified multimodal self-improvement.
  • Risks: Easier RL optimization may amplify text-to-image risks, including misleading visual content, protected-style imitation, and unsafe images.The passage specifically links these risks to stronger prompt following.
  • Risks: SpectraReward may inherit biases and visual misjudgments from its pretrained MLLM reward backbone during policy optimization.This risk arises because the reward is derived from the pretrained MLLM.
Loading 2607.11886v1…