PaperlayerView exact revision on arXiv

Source-linked AI summary

PromptRL: Prompt Matters in RL for Flow-Based Image Generation

Fu-Yun Wang, Han Zhang, Michael Gharbi, Hongsheng Li, Taesung Park

arXiv:2602.01382v1cs.CVcs.LG

TL;DR

Current flow-based RL pipelines face limited exploration diversity and prompt overfitting. PromptRL jointly trains language models as adaptive prompt refiners within flow-based RL, achieving state-of-the-art benchmark performance with up to 2× fewer rollouts.

  • Problem

    Flow-based RL pipelines exhibit exploration collapse from reduced generation diversity and severe prompt overfitting across semantically equivalent formulations.

  • Method

    PromptRL jointly trains language models as adaptive prompt refiners and flow models, generating semantically faithful prompt variations within the RL loop.

  • Results

    PromptRL achieves state-of-the-art performance across multiple benchmarks while requiring up to 2× fewer rollouts than existing methods.

  • Takeaways & Limitations

    Language-model-generated prompt variations expand exploration and improve RL sample efficiency while maintaining robust generalization to diverse prompt formulations.

  • Takeaways & Limitations

    The flow model partially co-adapts to its training-time language model, with GenEval dropping from 0.97 to 0.88 when the prompt enhancer is replaced.

Abstract

from arXiv · show

Flow matching models (FMs) have revolutionized text-to-image (T2I) generation, with reinforcement learning (RL) serving as a critical post-training strategy for alignment with reward objectives. In this research, we show that current RL pipelines for FMs suffer from two underappreciated yet important limitations: sample inefficiency due to insufficient generation diversity, and pronounced prompt overfitting, where models memorize specific training formulations and exhibit dramatic performance collapse when evaluated on semantically equivalent but stylistically varied prompts. We present PromptRL (Prompt Matters in RL for Flow-Based Image Generation), a framework that incorporates language models (LMs) as trainable prompt refinement agents directly within the flow-based RL optimization loop. This design yields two complementary benefits: rapid development of sophisticated prompt rewriting capabilities and, critically, a synergistic training regime that reshapes the optimization dynamics. PromptRL achieves state-of-the-art performance across multiple benchmarks, obtaining scores of 0.97 on GenEval, 0.98 on OCR accuracy, and 24.05 on PickScore. Furthermore, we validate the effectiveness of our RL approach on large-scale image editing models, improving the EditReward of FLUX.1-Kontext from 1.19 to 1.43 with only 0.06 million rollouts, surpassing Gemini 2.5 Flash Image (also known as Nano Banana), which scores 1.37, and achieving comparable performance with ReasonNet (1.44), which relied on fine-grained data annotations along with a complex multi-stage training. Our extensive experiments empirically demonstrate that PromptRL consistently achieves higher performance ceilings while requiring over 2$\times$ fewer rollouts compared to naive flow-only RL. Our code is available at https://github.com/G-U-N/UniRL.

1. Introduction

PromptRL addresses diversity loss and prompt overfitting in flow-based RL by jointly training language models as adaptive prompt-refinement agents with flow-matching generators. This co-training expands exploration while preserving semantic intent and achieves strong benchmark and image-editing results.

  • Limitations: Flow-based RL suffers from reduced generation diversity as prompt adherence improves, causing exploration to concentrate in narrow modes and optimization to stagnate.The introduction identifies this as an underappreciated exploration paradox in current pipelines.
  • Motivation: Existing methods treat prompts as fixed inputs, while random synonym substitution and rule-based paraphrasing fail to produce coherent variations at scale.The paper frames fixed-prompt optimization as a fundamental design oversight.
  • PromptRL: PromptRL jointly trains language models within flow-based RL loops to generate semantically grounded prompt variations that expand exploration and improve downstream rewards.The language models act as adaptive co-learners rather than static preprocessors, creating a mutually beneficial training dynamic for prompt and image generation.
  • Results: 0.97 on GenEval, 0.98 on OCR accuracy, and 24.05 on PickScore are PromptRL’s reported benchmark scores.These results are presented as state-of-the-art performance across multiple benchmarks.
  • Results: 1.19 to 1.43 EditReward improvement is achieved for FLUX.1-Kontext with only 0.06 million rollouts.The introduction reports this as validation of the RL approach on large-scale image editing models.

2. Related works

Related work spans reinforcement-learning methods for flow-based image generation and language-model-driven prompt enhancement. These studies motivate prompt refinement as a way to address the quality-diversity tradeoff affecting exploration in flow-based T2I optimization.

  • RL for image generation: RL for flow-based image generation includes differentiable reward methods and RL-based approaches that optimize generation using reward objectives.Differentiable methods such as DRaFT, AlignProp, and ReFL backpropagate pretrained reward gradients but can suffer reward hacking, including oversaturation; DDPO and DPOK represent RL-based approaches.
  • PE for image generation: Prompt enhancement has progressed from manual refinement to language-model-based automated prompt optimization for improving T2I quality and alignment.Promptist combines supervised fine-tuning with RL to optimize prompts for aesthetic appeal while preserving user intent.
  • Quality-diversity tradeoff: As flow-based models improve alignment and aesthetic quality, their output diversity can decline, creating an exploration bottleneck that prompt refinement partially mitigates.Figure 2 describes LM-based prompt refinement as restoring diversity while maintaining quality.
  • PE for image generation: Recent prompt-enhancement methods explore constrained decoding, iterative refinement, chain-of-thought reasoning, and reward-guided training for controllable or structured reprompting.NeuroPrompts supports user-controllable styles, OPT2I maximizes consistency scores, and RePrompt uses chain-of-thought reasoning with reward-guided training.

3. Understanding flow RL inefficiencies

Flow-based RL faces a quality–diversity tradeoff that limits exploration and degrades reward learning, while prompt linguistic hacking causes severe overfitting to superficial formulations. Paraphrase evaluation exposes major performance collapse in flow-only RL models despite pretrained models’ linguistic robustness.

  • Quality–diversity dilemma: As flow-based models become more prompt-aligned, they sacrifice output diversity, creating a fundamental tension between generation quality and effective RL exploration.The study measures prompt alignment with TI-Sim, aesthetic quality with PickScore, and image diversity with II-Sim.
  • Quality–diversity dilemma: Deterministic generation collapses rollout trajectories into narrow modes, causing reward signals and advantage estimates to lose comparative information for policy improvement.When samples cluster around similar high-quality outputs, advantage estimators cannot provide the contrast needed for learning.
  • Prompt overfitting: RL-trained models exhibit prompt linguistic hacking by exploiting superficial lexical patterns instead of developing robust semantic understanding.The evaluation compares original prompts with semantically preserved paraphrases generated by Qwen-2.5-VL.
  • Prompt overfitting: 0.92 to 0.81: FlowGRPO’s GenEval score drops under paraphrasing, demonstrating severe prompt overfitting.DiffusionNFT likewise performs strongly on original prompts but suffers catastrophic degradation after paraphrasing, whereas pretrained SD3 remains consistent or improves across metrics.

4. PromptRL

PromptRL integrates trainable language-model prompt refinement into flow-based RL, jointly optimizing prompt diversity and image generation while retaining baseline prompts. Its group-wise and reward-tagged training mechanisms stabilize optimization across prompts and heterogeneous objectives.

  • 4.1. Incorporating LMs as dynamic prompt refiner: PromptRL uses pretrained language models as adaptive prompt refiners, generating semantically faithful and linguistically diverse variants instead of relying on static synonym or rule-based augmentation.The refiners operate directly within the RL training loop and leverage pretrained LMs’ semantic and compositional flexibility.
  • 4.1. Incorporating LMs as dynamic prompt refiner: Each refined prompt is paired with independent noise and processed by the flow-matching model, creating hierarchical exploration through both linguistic and latent variation.For an original prompt p0, the LM generates {p1, p2, . . . , pk}, which produce diverse samples through πFM(·|pi, ϵi).
  • 4.1. Incorporating LMs as dynamic prompt refiner: Prompt retention keeps m < n original-prompt samples in every batch, providing a baseline for pruning lower-reward refinements and preserving stable optimization.The remaining n−m samples use LM-based augmentation, while unmodified prompts support advantage estimation.
  • 4.2. Joint RL training on disjoint LMs and FMs: PromptRL jointly trains πLM and πFM with shared rewards but separate policy gradients, keeping the models architecturally disjoint and computationally modular.The framework requires no architectural or algorithmic modifications and can use GRPO or other online RL approaches.
  • 4.2. Joint RL training on disjoint LMs and FMs: Group-wise reward normalization makes advantages invariant to prompt-specific reward scales while inducing self-competition among samples from the same prompt.Within each group, rewards are normalized using its mean μj and standard deviation σj, with a small numerical-stability constant ϵ.
  • 4.2. Joint RL training on disjoint LMs and FMs: The LM updates only on refined samples, whereas the FM updates on all original and refined samples, preserving base-distribution performance while rewarding beneficial prompt changes.Underperforming variants receive negative advantages and are down-weighted; original prompts remain included in FM optimization.
  • 4.3. Multi-reward training via reward tagging: Reward tagging enables multi-reward training with GenEval, PickScore, and OCR by normalizing advantages within categories, eliminating manual reward-coefficient tuning.Each prompt receives a categorical tag specifying which reward evaluates its generated images, allowing objectives to remain in their native scales.

5. Experiments

PromptRL is evaluated on text-to-image generation and instructional image editing using FLUX flow-matching models with Qwen2.5-VL-3B-Instruct as the prompt refiner. Across compositionality, OCR, preference alignment, editing, multi-reward training, and efficiency tests, joint LM-FM optimization delivers strong performance with fewer rollouts.

  • Text-to-image generation: PromptRL achieves 0.97 on GenEval, surpassing FlowGRPO at 0.92 and DiffusionNFT at 0.88.PromptRL w/ PE reaches 0.99 on both Position and Counting, while PromptRL w/o PE achieves 0.94.
  • Text-to-image generation: 0.98 OCR accuracy on OCR-1k and 32.03 on the accompanying aesthetic metric surpass prior RL-based approaches.The gains across diverse metrics indicate that joint LM-FM optimization extends beyond single-objective reward optimization.
  • Instructional image editing: 1.43 EditReward improves on the FLUX.1-Kontext baseline at 1.19 and approaches ReasonEdit-Think at 1.44.The largest category gains are Removal (+0.69) and Environment (+0.28); naive prompt enhancement without joint training degrades baseline performance.
  • Multi-reward training: 0.93 GenEval, 0.96 OCR, and 23.94 PickScore demonstrate competitive multi-reward performance without coefficient tuning or a multi-stage curriculum.The multi-reward model shows only modest degradation relative to single-reward specialists.
  • Training efficiency: Approximately 50% fewer rollouts are needed for PromptRL to reach FlowGRPO’s GenEval convergence point.PromptRL consistently achieves higher rewards with fewer rollouts across both GenEval and OCR under the stated training setup.
  • Prompt retention mechanism: 0.83 GenEval results when retaining no original prompts and 0.76 when retaining one show degraded performance on unmodified test prompts.With group size n = 8, the study varies retained original prompts m ∈ {0, 1, 2, 4} and attributes the degradation to rapid discovery of FM-suitable prompt variants.

6. Conclusion

PromptRL jointly trains language and flow-matching models in a unified reinforcement learning loop for text-to-image generation. It expands exploration through LM-generated prompt variations, improves inference-time generation quality via a co-evolving prompt enhancement module, and achieves state-of-the-art benchmark performance.

  • PromptRL jointly trains language models and flow-matching models within a unified reinforcement learning loop for text-to-image generation.
  • LM-generated prompt variations expand the exploration space while a co-evolving prompt enhancement module improves generation quality at inference time.
  • PromptRL achieves state-of-the-art performance across multiple benchmarks.

A. Training details

PromptRL training alternates LM prompt refinement, FM image generation, and updates to both models using group-normalized advantages. Its implementation uses prompt retention, selective gradient updates, consistent configurations with task-specific adjustments, and higher-resolution image editing settings to balance quality and efficiency.

  • A. Training details: PromptRL alternates LM prompt-variant generation, FM image production, and updates to both models using group-normalized advantages.The procedure is presented in Algorithm 1.
  • A. Training details: The prompt retention mechanism maintains m original prompts per group, while selective gradients train the LM on refined prompts and the FM on all samples.These mechanisms are key design choices in the training procedure.
  • A. Training details: Training configurations remain consistent where possible, with task-specific learning rates and KL coefficients; image editing uses 1024×1024 resolution and fewer SDE steps.The image-editing settings balance quality and efficiency.

B. Discussion

PromptRL generalizes to unseen flow models and outperforms flow-only RL even when the latter uses twice the rollouts, supporting joint LM-FM optimization while revealing co-adaptation and the importance of prompt phrasing. Its gains are attributed to linguistic diversity that helps flow models escape narrow reward modes.

  • Generalization of prompt enhancer to unseen FMs: PromptRL’s prompt enhancer significantly improves unseen flow models SANA and SD3, although it does not match model-specific prompt-only RL enhancers.Evaluation uses GenEval at 1024 resolution with 20 inference steps.
  • Comparison to flow-only RL: 0.93 vs. 0.97 GenEval, 0.93 vs. 0.98 OCR, and 23.85 vs. 24.05 PickScore: flow-only RL underperforms PromptRL despite twice the rollouts.The comparison controls for computational budget by giving flow-only RL 2× the number of rollouts.
  • Comparison to flow-only RL: PromptRL injects linguistic diversity, enabling the flow model to escape narrow reward modes and achieve higher performance ceilings.This mechanism is presented as an explanation for PromptRL’s advantage over flow-only RL.
  • Limitations: Replacing the co-trained prompt enhancer with Qwen-3 at inference drops GenEval from 0.97 to 0.88, indicating FM specialization to its training-time LM partner.The drop reflects co-adaptation between the flow model and language model during joint optimization.
  • Why JointRL matters?: Small linguistic prompt changes can substantially affect generation quality, making prompt enhancement as important as the flow model and motivating joint optimization.The passage argues that prompt refinement depends on the current flow model, and vice versa.
Loading 2602.01382v1…