PaperlayerView exact revision on arXiv

Source-linked AI summary

RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time

Haozhe Wang, Cong Wei, Weiming Ren, Jiaming Liu, Fangzhen Lin, Wenhu Chen

arXiv:2604.11626v2cs.AIcs.LG

TL;DR

Visual-generation reward models often reduce multidimensional human judgments to unexplained scalar scores. The paper introduces RationalRewards, trained with PARROT to generate structured rationales before scoring, and uses those rationales for reinforcement learning and test-time prompt refinement. RationalRewards improves generators beyond scalar alternatives, while its no-update critique-and-refine loop matches or exceeds RL-based fine-tuning on several benchmarks.

  • Problem

    Most visual-generation reward models compress multidimensional human judgments into unexplained scalar scores, discarding the reasoning underlying preference.

  • Method

    RationalRewards generates structured, multidimensional rationales before scores, while PARROT recovers latent rationales from pairwise preference data.

  • Results

    RationalRewards improves text-to-image and image-editing generators beyond scalar baselines, and its no-update Generate–Critique–Refine loop matches or exceeds RL-based fine-tuning on several benchmarks.

  • Takeaways & Limitations

    Structured critiques provide an optimization interface across parameter and prompt spaces, with test-time refinement offering an alternative to parameter updates.

  • Takeaways & Limitations

    The latent-capability explanation remains a hypothesis requiring representation-level validation.

Abstract

from arXiv · show

Most reward models for visual generation reduce rich human judgments to a single unexplained score, discarding the reasoning that underlies preference. We show that teaching reward models to produce explicit, multi-dimensional critiques before scoring transforms them from passive evaluators into active optimization tools, improving generators in two complementary ways: at training time, structured rationales provide interpretable, fine-grained rewards for reinforcement learning; at test time, a Generate-Critique-Refine loop turns critiques into targeted prompt revisions that improve outputs without any parameter updates. To train such a reward model without costly rationale annotations, we introduce Preference-Anchored Rationalization (PARROT), a principled framework that recovers high-quality rationales from readily available preference data through anchored generation, consistency filtering, and distillation. The resulting model, RationalRewards (8B), achieves state-of-the-art preference prediction among open-source reward models, competitive with Gemini-2.5-Pro, while using 10-20x less training data than comparable baselines. As an RL reward, it consistently improves text-to-image and image-editing generators beyond scalar alternatives. Most strikingly, its test-time critique-and-refine loop matches or exceeds RL-based fine-tuning on several benchmarks, suggesting that structured reasoning can unlock latent capabilities in existing generators that suboptimal prompts fail to elicit.

1 introduction

RationalRewards addresses scalar reward models’ loss of structured human judgment by producing explicit critiques before scores, enabling optimization during both training and inference. PARROT derives such rationales from preference data, and the resulting model improves reinforcement learning while its critique-and-refine loop can match or surpass RL-based optimization without parameter updates.

  • Motivation: Most visual-generation reward models compress perceptual quality, instruction faithfulness, physical plausibility, and text rendering into one unexplained score.This discards structured preference reasoning and can leave generators exploiting shortcut correlations.
  • Contribution: RationalRewards generates structured, multi-dimensional critiques before deriving scores, turning reward modeling into an optimization interface.Its rationales provide semantically grounded feedback for downstream generation improvement.
  • Dual-space optimization: In parameter space, RationalRewards supplies dense, interpretable reinforcement-learning feedback that replaces opaque scalar gradients prone to reward hacking.The rationales explain what to improve and why.
  • Dual-space optimization: In prompt space, its postgeneration Generate–Critique–Refine loop identifies image deficiencies and converts them into targeted prompt revisions without parameter updates.This trades test-time computation for improved fidelity.
  • Training: PARROT treats rationales as latent variables and learns them from widely available pairwise preference data rather than costly rationale annotations.The framework derives an evidence lower bound on observed preferences.
  • Results: RationalRewards achieves state-of-the-art preference prediction among open-source reward models, improves generators beyond scalar baselines, and matches or exceeds RL-based fine-tuning on several benchmarks.The test-time result requires no parameter updates.

2 Method

RationalRewards trains reward models to generate preference-grounded, multi-dimensional rationales before scoring, then uses those rationales for parameter-space and prompt-space optimization. PARROT recovers rationales from preference data through anchored generation, consistency filtering, and distillation.

  • Reward-model design: RationalRewards generates structured critiques across dimensions including text faithfulness, image faithfulness, physical quality, and text rendering before producing scores.For image editing, the assessment taxonomy also includes image faithfulness.
  • PARROT training: PARROT treats rationales as latent variables and decomposes its ELBO into rationale generation, predictive-consistency filtering, and student distillation.The framework uses pairwise preference data without requiring ground-truth rationale annotations.
  • Optimization: RationalRewards maintains reward–quality correspondence and improves resistance to reward hacking by requiring coherent, multi-dimensional justifications for scores.The paper contrasts this with scalar rewards, whose increases can accompany visibly degraded generation quality.
  • PARROT training: The student is trained to predict rationales and preferences from inputs alone, using filtered posterior samples through the hindsight–foresight teacher–student decomposition.Hindsight infers rationales with known preferences; foresight predicts rationales and preferences without them.
  • PARROT training: Preference anchoring conditions teacher-generated rationales on known preference labels, producing posterior samples more consistent with observed human judgments.A consensus check removes rationales that cannot recover the preference without the label.
  • Pointwise deployment: Pointwise projection adapts pairwise training to individual-image scoring by prompting the teacher to assign 1–4 scores across four evaluation dimensions.Validated pairwise rationales guide the teacher’s attention toward identified defects, and the student is trained jointly on pairwise and pointwise data.
  • Optimization: Despite being 4× smaller, RationalRewards consistently outperforms the generic Qwen3-VL-32B judge as an RL reward signal across tested generators.The paper attributes this advantage to preference training that calibrates more stable, preference-aligned pointwise scores.
  • Optimization: RationalRewards supports parameter-space optimization with decomposed RL rewards and prompt-space optimization through targeted Generate–Critique–Refine revisions.The test-time loop uses critiques to identify concrete deficiencies and revise prompts without parameter updates.

3 Experiments

The experiments evaluate RationalRewards as a preference evaluator and as a dual-space optimizer for visual generation. Across editing and text-to-image tasks, its preference-trained structured rewards improve RL, while test-time prompt refinement adds further gains.

  • Experimental Setup: 30K editing pairs and 50K text-to-image pairs provide preference-only training data for RationalRewards.The datasets contain binary or ranked preferences without rationale annotations.
  • Preference Modeling: RationalRewards surpasses open-source scalar reward models across three preference-prediction benchmarks and approaches larger commercial models.The 8B model outperforms Gemini-2.5-Flash and approaches GPT-5/Gemini-2.5-Pro.
  • Preference Modeling: 6.8 points on MMRB2 (T2I) and 17.3 points on GenAI (Edit) separate RationalRewards from direct distillation using the same data volume.This ablation isolates preference-anchored rationalization from generic knowledge distillation.
  • Parameter-Space Tuning: 3.84 versus 3.66 on ImgEdit-Bench shows RationalRewards-guided RL outperforming EditReward-guided RL for Flux.1 Kontext.RationalRewards improves Flux.1 Kontext from a 3.52 base score to 3.84 overall.
  • Parameter-Space Tuning: 70.34 versus 62.55 lifts FLUX.1-dev on UniGen, exceeding both MultiReward and direct Qwen3-VL-32B judging.The improvement over the base is +9.37 points, and the direct Qwen3-VL-32B judge reaches 66.71.
  • Test-Time Prompt Tuning: 8.33 versus 8.29 shows prompt tuning slightly exceeding RL alone on GEdit-Bench-EN Overall.On ImgEdit-Bench, prompt tuning further boosts the RL-tuned Flux model from 3.84 to 4.01.
  • Interpretation and Caveat: LoRA update capacity and RL query-distribution coverage constrain the reported RL ceiling, whereas prompt tuning performs per-instance optimization without catastrophic forgetting.The paper presents latent generator capability as a hypothesis requiring representation-level validation.

4 Related Work

Prior visual-generation work relies heavily on scalar reward models and has increasingly used reinforcement learning to improve generation. These approaches make reward quality and reward hacking central concerns.

  • Reward Models for Visual Generation: Scalar reward models commonly map visual inputs directly to opaque scores from large-scale human preference datasets.Examples include ImageReward, VideoReward, PickScore, UnifiedReward, and EditReward.
  • Training and Test-Time Scaling: Recent visual-generation systems integrate reinforcement learning to improve capabilities such as compositional reasoning and text rendering.The cited methods include FlowGRPO, DanceGRPO, Blip3o-Next, and DiffusionNFT.
  • Training and Test-Time Scaling: RL remains bottlenecked by reward-model quality and can suffer reward hacking when proxy rewards diverge from the intended objective.This motivates alternatives to scalar reward supervision for visual generation.

5 Conclusions

The paper concludes that structured rationalization makes an 8B reward model effective for preference prediction and generation optimization. Its test-time critique-and-refine loop can match or exceed RL fine-tuning without parameter updates.

  • Conclusions: An 8B RationalRewards model achieves preference-prediction accuracy competitive with Gemini-2.5-Pro while using 10–20× less training data than scalar baselines.The conclusion describes structured rationalization as an inductive bias for preference modeling.
  • Conclusions: Structured multi-dimensional rationales consistently outperform scalar reward models and larger generic VLM judges as RL rewards across text-to-image and image-editing benchmarks.The conclusion attributes the result to semantically grounded rewards.
  • Broader Uses: RationalRewards supports explainable data curation, multi-dimensional reward modeling, preference-calibrated prompt tuning, and regional flaw grounding.These roles are summarized in Figure 8.
  • Conclusions: The Generate–Critique–Refine loop matches or exceeds RL-based fine-tuning on several benchmarks without parameter updates.The result supports, but does not establish, the hypothesis that suboptimal prompts fail to elicit latent generator capabilities.

A Extended Experimental Results

Extended experiments examine benchmark coverage, training behavior, reward stability, out-of-distribution physics-aware editing, and regional critique grounding. The reported analyses emphasize stable rewards and localized visual diagnostics.

  • Extended Benchmarks: Tables 4–6 extend text-to-image, generic image-editing, and physics-aware PICA-Bench results across categories and model variants.PICA-Bench provides out-of-distribution physics-aware editing evaluation.
  • Training Dynamics: Training curves and qualitative visualizations document RationalRewards behavior throughout RL training.The appendix references Figure 10 for qualitative results and Figure 9 for training curves.
  • Reward Stability: RationalRewards shows smooth, converging reward and standard-deviation curves, while EditReward has high variance and MultiReward poorly differentiates high-capability generations.The comparison is presented as evidence about reward stability and discrimination.
  • Visual Grounding: Regional flaw grounding uses structured referring expressions, GroundingDINO bounding boxes, and SAM segmentation masks to localize problematic image regions.This provides an additional diagnostic and dense-reward use case.

B ELBO Derivation and Theoretical Details

The ELBO derivation models preferences with latent rationales, then decomposes optimization into prediction, regularization, and pipeline-aligned phases. The appendix also states assumptions behind extending pairwise rationales to pointwise scores and identifies two failure modes.

  • ELBO formulation: The model introduces a latent rationale z explaining preference y for image pair and conditioning request x.The observed input contains two generated images and a user request.
  • ELBO formulation: Because marginalizing over natural-language rationales is intractable, PARROT introduces a variational posterior qϕ(z | x, y).The posterior conditions on both the input and known preference.
  • ELBO decomposition: Applying Jensen’s inequality yields an ELBO whose integrand contains preference prediction, rationale likelihood, and posterior regularization terms.The decomposition is written using log Pθ(y | x, z), log Pθ(z | x), and −log qϕ(z | x, y).
  • Pipeline interpretation: The three ELBO terms map to rationale generation, consistency filtering, and foresight distillation in the PARROT pipeline.Filtering retains rationales whose preference can be recovered from the input and rationale, while distillation trains the student on filtered samples.
  • Assumptions: The derivation assumes Pθ(z, y | x) = Pθ(y | x, z) · Pθ(z | x), treating the rationale as mediating preference prediction.This autoregressive factorization is described as natural for language models.
  • Pointwise projection: Pointwise projection assumes pairwise and absolute assessments share evaluation principles, using validated pairwise rationales as reference hints.The teacher independently assigns float-valued 1–4 scores across four dimensions.
  • Limitations: Pointwise projection may suffer calibration drift or dependence on the comparison partner despite rubric anchors and independent-assessment instructions.These are acknowledged as potential failure modes rather than reported failures in the supplied passage.

E Implementation Details

The implementation uses DiffusionNFT for parameter-space RL, aggregating RationalRewards’ applicable dimension scores into rewards. Training uses LoRA and low-reward prompts, while inference applies a thresholded refinement pass.

  • RL training: DiffusionNFT samples K images, ranks them by reward, and updates the diffusion policy with a contrastive flow-matching loss.High-reward generations form implicit positives and low-reward generations form negatives.
  • RL training: RationalRewards supplies per-dimension scores that are averaged equally across applicable dimensions for the RL reward.Image Faithfulness is excluded for text-to-image tasks, and Text Rendering is excluded when no text is generated.
  • RL training: LoRA provides parameter-efficient fine-tuning on a distributed setup using 16 NVIDIA A100-80GB GPUs.Eight GPUs train the model and eight serve the reward model for online evaluation.
  • RL training: RL prompts are selected from EditReward and HPDv3 when initial generations score below 3.0 on average.This focuses training on prompts where the generator has more room for improvement.
  • Inference: The test-time refinement pass regenerates an image when any dimension score falls below 3.0, adding approximately 0.4 seconds of VLM inference overhead.The threshold is the midpoint of the 1–4 scoring scale.

F.1 Training Data Statistics

The PARROT pipeline uses substantially less preference data than comparable baselines, with filtering reducing the raw pair count before projection samples are created.

  • Training data: Approximately 80K raw preference pairs become 57.6K after consistency filtering.Each pairwise sample yields two pointwise projection samples, one per image.
  • Training data: EditReward uses 200K pairs, while UnifiedReward uses over 1M pairs, compared with PARROT’s approximately 80K raw pairs.The paper attributes part of this data efficiency to distilling the teacher’s pretrained knowledge through structured rationales.

F.2 Consistency Filtering Analysis

Consistency filtering retains most generated rationales but removes hallucinated, preference-inconsistent, and nondiscriminative explanations.

  • Filtering outcome: Approximately 72% of generated rationales are retained after consistency filtering.Rejected rationales exhibit several recurring failure modes.
  • Rejected rationales: Visual hallucinations describe content absent from the images and can produce incorrect preference predictions when the label hint is removed.The passage gives an invented sunset as an example.
  • Rejected rationales: Label-ignoring rationales favor the non-preferred image, especially when quality differences are subtle.This failure occurs despite preference anchoring.
  • Rejected rationales: Vague rationales provide generic praise or criticism without enough discriminative detail to distinguish the images.Such explanations cannot reliably support preference prediction.

G.1 Limitations

The paper identifies limitations involving teacher dependence, inherited bias, unvalidated explanations for latent capability, threshold selection, and evaluation scope.

  • Teacher Model Dependence: RationalRewards’ quality is upper-bounded by its teacher model, whose blind spots can be inherited in specialized domains.The paper specifically cites fine-grained physics simulation, culturally specific aesthetics, and specialized technical content.
  • Bias Inheritance: Preference datasets and teacher pretraining encode biases that may favor certain visual styles, demographics, or content types.The authors have not conducted a comprehensive bias audit and recommend evaluation on diverse, underrepresented content.
  • Latent Capability Hypothesis: The latent-capability explanation for test-time prompt tuning remains unvalidated at the representation level, and alternative explanations cannot be ruled out.One alternative is that prompt refinement simply provides additional context that benefits models generally.
  • Threshold Sensitivity: The Generate–Critique–Refine loop uses a fixed 3.0 refinement threshold without comprehensive sensitivity analysis across benchmarks and generators.The optimal threshold may vary with generator capability and task difficulty.
  • Language and Domain Scope: All evaluations use English-language benchmarks, leaving transferability to other languages and non-photorealistic domains untested.Un tested domains include 3D rendering, video generation, and scientific visualization.

G.2 Broader Impact

The paper discusses broader benefits of RationalRewards, including accessible evaluation, interpretable critiques, and easier prompt refinement, alongside misuse and bias risks requiring responsible deployment.

  • Positive Impacts: An open-source reward model competitive with commercial alternatives could lower barriers to evaluating visual-generation quality without costly proprietary APIs.The paper frames this as democratized evaluation for researchers and practitioners.
  • Positive Impacts: Structured, multi-dimensional critiques can expose specific visual failure modes instead of requiring optimization against opaque scalar scores.This interpretability may help users and developers understand and address quality problems.
  • Positive Impacts: The Generate–Critique–Refine loop can help users with limited prompt-engineering experience achieve higher-quality generations.It automatically identifies deficiencies in instructions and addresses them through refinement.
  • Negative Impacts and Mitigations: Improved image-generation quality could be misused to create misleading visual content, deepfakes, or other harmful media.Although RationalRewards does not generate images, its use as an RL reward or prompt optimizer could amplify generator capabilities.
  • Negative Impacts and Mitigations: Biases in preference data may cause reward models to favor certain content types and amplify disparities in visual representation.The paper recommends domain-specific evaluations before production deployment, especially in sensitive applications.
Loading 2604.11626v2…