Think Before You Score: Thinking Reward Model for Visual Generation

• Conventional denoising and flow-matching objectives do not explicitly capture human preferences, while direct scalar scoring leaves its evaluation process implicit.Preference-based reward signals are increasingly used in reinforcement learning to align generation with human preferences.Source: §1 · p. 1
• Because evaluation criteria and failure modes vary across cases, even reasoning-based scoring leaves open what should be evaluated for a particular case.The authors frame identifying case-relevant criteria as an underexplored question.Source: §1 · p. 2
• TRM formulates a case-adaptive rubric, assesses the candidate against each criterion, integrates the evidence, and outputs a pointwise reward.The rubric connects task interpretation with quality assessment in a structured evaluation process.Source: §1 · p. 2
• A Bradley–Terry-style objective applied directly to pointwise scores can keep enlarging score margins after preference ordering is correct, risking polarized score distributions.PD-GRPO instead uses response-level relative optimization while retaining fine-grained pointwise scoring.Source: §1 · pp. 2–3
• Think Before You Score determines what matters for each case before judging how well its candidate performs.The principle shares the process of deriving and applying criteria across tasks while adapting the criteria to the case.Source: §1 · pp. 2–3
• The authors build image-generation and editing training data with rubric, criterion-assessment, and quality-score supervision, then train TRM with cold-start SFT followed by PD-GRPO.PD-GRPO uses pairwise preferences to improve fine-grained pointwise discrimination while mitigating score polarization.Source: §1 · p. 3
• Experiments show TRM performs strongly on image-generation and editing reward benchmarks and provides effective rewards for improving diverse generation models through reinforcement learning.The reported results cover both reward modeling and downstream generation optimization.Source: §1 · p. 3

• Existing visual reward models largely rely on fixed evaluation criteria despite variation in requirements and potential failure modes across cases.Table 1 compares representative reward models.Source: §2 · p. 3, Table 1 · p. 3
• Prior visual reward models use regressive scalar prediction [48] [50] or generative multimodal assessment [32] [35] [56], increasingly with reasoning before scoring [26] [34] [40].Pointwise and pairwise evaluation are both used; generation commonly considers prompt adherence and visual quality, while editing also considers edit correctness and content preservation.Source: §2 · p. 3
• Prior reinforcement-learning work optimizes visual generation with reward feedback, while this work focuses on strengthening that feedback through pairwise supervision and applying it to generation and editing.The reviewed approaches include Flow-GRPO [23] and subsequent methods for sampling efficiency, policy constraints, and preference-based advantage estimation.Source: §2 · pp. 3–4

3.1 Can Reward Modeling Be Unified Across Visual Generation Tasks?

• Visual generation cases can be represented as a task condition and candidate output, with a shared goal of estimating how well the output satisfies that condition.For generation the condition is a text prompt; for editing it is a source image plus an editing instruction.Source: §3.1 · p. 4

3.2 How Should a General Reward Model Evaluate a Visual Generation Case?

• Evaluation criteria should adapt to both task requirements and candidate-specific issues, which direct scalar-reward models leave implicit.Examples include object relations or visual defects in generation, and identity preservation or unintended changes in editing.Source: §3.2 · p. 4
• TRM structures evaluation as R → J → H → s: instantiate a case-adaptive rubric, inspect and judge against it, summarize dimension-level assessments, then produce a pointwise reward.The rubric is an evaluation plan generated from the task condition and candidate before the corresponding judgments and final reward.Source: §3.2 · p. 4
• TRM uses Prompt Alignment and Visual Quality for both tasks, with Aesthetics for generation and Source Consistency for editing, and adapts atomic rubrics within each dimension.Rubric checks yield binary judgments, which are summarized into dimension-level assessments before the final reward is predicted.Source: §3.2 · p. 4

4.1 Data Construction and Cold-Start SFT

• Raw cases are filtered for reliability, expanded with benchmark data and model rollouts, and balanced across tasks, sources, and difficulty levels.The pipeline covers diverse task categories, quality levels, and failure modes.Source: §4.1 · p. 5
• Two-stage human–AI annotation has experts refine prompts and review case-adaptive rubrics, then uses verified rubrics for judgments and scores followed by expert calibration.The resulting dataset contains approximately 20K image generation cases and 28K image editing cases.Source: §4.1 · p. 5
• Cold-start SFT jointly trains rubric generation, rubric-based scoring, and integrated evaluation to learn the complete rubric-then-score process.TRM (SFT) initializes subsequent reinforcement learning.Source: §4.1 · p. 5
• Preference pairs are ranked by reward gap into easy, medium, and hard subsets, then balanced across task categories, candidate sources, and difficulty levels.The resulting set contains approximately 4,000 pairs.Source: §4.1 · p. 5

4.2 Pairwise Preference Optimization

• PD-GRPO rewards each rollout when its pointwise score clears a margin against the opposite rollout group, with no added reward for widening an already sufficient gap.The margin can be adjusted to preference-pair difficulty.Source: §4.1 · p. 5, §4.2 · p. 6
• Unlike the BT-style reward, which can drive preferred and dispreferred scores toward opposite boundaries, PD-GRPO stops rewarding score-gap expansion after the required separation.The BT-style objective has no interior optimum with respect to the score gap.Source: §4.2 · p. 6
• Candidates interact only during reward construction, so PD-GRPO uses pairwise training supervision while preserving independent pointwise evaluation at inference.Rollouts produce structured evaluations and scores without observing the other candidate.Source: §4.2 · pp. 6–7

5.1 Experimental Setups

• The experiments assess reward-modeling performance on image generation and editing benchmarks and test downstream RL on generation and editing models.Reward-model comparisons include proprietary, open-source, and specialized models; downstream evaluations use multiple model families and benchmarks.Source: §5.1 · p. 7

5.2 Reward Model Performance

• 71.2%/67.9% on GenAI-T2I/MMRB2-T2I are TRM’s final image-generation results, surpassing GPT-4.1 on both benchmarks and all evaluated open-source and specialized models on MMRB2-T2I.Cold-start SFT reaches 70.1%/65.8% from the Qwen-3.5 baseline’s 58.9%/59.4%.Source: Table 2 · p. 7, §5.2 · p. 8
• 58.2% on MMRB2 and 0.640/0.660 on EditReward-Compass are among the editing gains from PD-GRPO; 9B TRM (RL) also outperforms the specialized 72B EditScore model across reported benchmarks and metrics.TRM (SFT) reaches 0.750/0.639/0.743 on EditScore-ERB and 67.8% on EditReward-ERB.Source: Table 3 · p. 8, §5.2 · p. 8

5.3 Reward-Guided RL Optimization

• 6.76 and 6.56 points on TIIF-Short and TIIF-Long, respectively, are FLUX.1-dev’s gains; TRM-guided RL consistently improves BAGEL, FLUX.1-dev, and SD3.5-M across evaluated generation benchmarks.BAGEL improves by 5.81 points on TIIF-Long; qualitative comparisons show better fine-grained constraint following, including object counts and text rendering.Source: §5.3 · pp. 8–9, Table 4 · p. 9
• 3.37→3.91 on BAGEL ImgEdit accompanies gains across nearly all editing metrics, model families, and scales; Figure 3 shows better preservation of unedited content and fewer unintended changes after SenseNova-U1.5 fine-tuning.SenseNova-U1.5 improves from 4.30 to 4.52 on ImgEdit and from 8.20/8.19 to 8.33/8.34 on GEdit-Bench-EN/CN.Source: Table 5 · p. 9, §5.3 · pp. 9–10, Figure 3 · p. 10

5.4 Ablation Studies

• TRM exceeds AlphaGRPO with the same BAGEL backbone by 2.98 and 3.33 points on TIIF-Short and TIIF-Long, with gains also reported on GenEval and DPG-Bench.Source: §5.4 · p. 10, Table 6 · p. 11
• At training step 300, reward standard deviation falls 14.9% with TRM versus 50.5% with EditScore-72B, retaining more variation among candidates for group-relative advantage estimation.TRM uses 9B parameters, one-eighth of EditScore-72B, and achieves comparable overall performance with slightly higher results on ImgEdit and GEdit-Bench-EN.Source: §5.4 · pp. 10–11, Table 6 · p. 11

• TRM constructs case-adaptive rubrics before scoring, using flexible case-level criteria within a shared evaluation process.Source: §6 · p. 11
• Structured SFT followed by PD-GRPO trains TRM to improve fine-grained reward discrimination while mitigating score polarization.Source: §6 · p. 11
• Experiments report strong reward-modeling performance on image generation and editing and consistent RL improvements across diverse generative-model architectures and scales.Source: §6 · p. 11

A.1.1 Supervised Fine-Tuning Data

• Approximately 48K SFT cases comprise around 20K image-generation and 28K image-editing cases, processed through a two-stage annotation pipeline for structured supervision.Source: §A.1.1 · p. 16
• Generation data cover prompt alignment, perceptual quality, structural coherence, and fine-grained defects, with focused human-centric, poster, multilingual-text, and reasoning subsets.The five taxonomy groups balance data categories; they do not function as fixed evaluation rubrics.Source: §A.1.1 · p. 16, Table 7 · p. 16
• Editing SFT combines filtered EditScore cases with curated benchmark tasks and candidate outputs spanning capability levels, including challenging spatial, interaction, perspective, and reasoning edits.Editing evaluation must consider instruction fulfillment, visual quality, and preservation of content outside the intended edit region.Source: §A.1.1 · pp. 16–17, Table 8 · p. 17
• The data are balanced across task categories, candidate sources, and quality levels; editing covers 19 fine-grained categories, while generation includes 20K cases across diverse categories.Intermediate-quality EditScore examples receive greater sampling emphasis, while high- and low-quality cases remain covered.Source: §A.1.1 · p. 17, Figure 4 · p. 18

A.1.2 Reinforcement Learning Data

• Pairwise preference data are constructed to improve TRM(SFT)’s fine-grained discrimination, including both clearly distinguishable candidates and close-quality pairs.Source: §A.1.2 · p. 18
• Generation preferences draw on OIP, EvalMuse, and HPDv3++, balancing prompts and candidate characteristics across easy, medium, and hard pairs according to score gaps.Source: §A.1.2 · p. 18
• Editing pairs compare outputs for the same source image and instruction across diverse tasks, assessing instruction fulfillment, visual fidelity, spatial correctness, and preservation of unrelated content.Sampling spans easy, medium, and hard comparisons, including candidates with similar quality.Source: §A.1.2 · p. 18

A.2.1 Supervised Fine-Tuning of the Reward Model

• TRM is fine-tuned in one stage on complete structured responses, jointly learning rubric generation, rubric-level evaluation, dimension-level reasoning, and final scoring.The data contain approximately 20K image-generation and 28K image-editing cases; image-generation final scores incorporate calibration from other evaluators, and splits are prompt-level.Source: §A.2.1 · p. 19

A.2.2 Reinforcement Learning of the Reward Model

• PD-GRPO trains from 4K preference pairs by independently sampling eight evaluation responses per candidate, preserving pointwise scoring while adding relative supervision.It uses hard-margin rewards on normalized scores, with m = 0 for editing and m = 0.05 for generation; validation focuses on 300 challenging pairs misranked by SFT.Source: §A.2.2 · p. 19

A.2.3 Reinforcement Learning for Image Generation

• FlowGRPO optimization uses TRM as the reward signal for BAGEL, FLUX.1-dev, and SD3.5-M, with LoRA training all three; image-editing models also use TRM and LoRA for parameter-efficient fine-tuning.Source: §A.2.3 · p. 20, §A.2.4 · p. 20

A.3.1 Relative Advantages of Pointwise and Pairwise Reward Modeling

• Pointwise scoring avoids quadratic exhaustive-comparison costs and supports larger rollout groups, while pairwise supervision improves relative quality discrimination during reward-model training.Order reversal exposes reliability issues: Qwen3.5-9B has a 54.7% inconsistency rate, and inconsistent pairs can leave aggregate accuracy unchanged while producing contradictory optimization feedback.Source: §A.3.1 · pp. 21–22

A.3.2 Analysis of Efficiency of TRM

• Asynchronous reward computation overlaps generation and evaluation, saving (K −1) min(tg, tr) under an idealized pipeline.The analysis assumes separate resources, constant processing times, and negligible communication overhead; it excludes the policy update.Source: §A.3.2 · p. 22

A.4.1 Qualitative Results of TRM-Guided Optimization

• TRM-guided optimization improves editing instruction following and content preservation, strengthens generation prompt adherence, and raises training and evaluation rewards while retaining variation among candidate scores.Qualitative examples cover diverse editing operations and fine-grained generation requirements; reward trends span image generation and editing.Source: §A.4.1 · p. 22, §A.4.2 · p. 23, Figure 8 · p. 25

A.4.3 Tie-Aware Evaluation for Image Generation

• Under tie-aware evaluation, TRM(RL) reaches pairwise preference accuracies of 68.4% on GenAI-T2I and 63.9% on MMRB2-T2I, versus Qwen3.5-9B (Baseline) at 54.6% and 53.5%, respectively.Source: Table 10 · p. 25, §A.4.3 · p. 25

A.5.1 Unified SFT System Prompts

• TRM’s unified response generates case-adaptive criteria, judges them, analyzes evidence by dimension, and produces a final pointwise score using task-specific prompts.Generation evaluates Prompt Alignment, Aesthetics, and Technical Quality; editing evaluates Instruction Following, Visual Consistency, and Visual Quality, with binary rubric judgments and a scalar score from 0 to 10.Source: §A.5.1 · pp. 25–26, System Prompt for the Image Generation… · p. 28

System Prompt for the Image Generation Reward Model

• For image generation, the prompt assesses Prompt Alignment, Aesthetics, and Technical Quality using positive, atomic, visually verifiable checklist criteria.It typically calls for 8–14 checklist points, decomposes prompt requirements, and scores each point 0 or 1, assigning 0 under uncertainty. The final score uses checklist judgments and important uncovered issues; the prompt anchors scores at 0, 5, and 8.Source: System Prompt for the Image Generation… · p. 28

System Prompt for the Image Editing Reward Model

• For image editing, the prompt assesses Instruction Following, Visual Consistency, and Visual Quality through positive, atomic criteria focused on the requested edit and unintended changes.Criteria are scored independently as 0 or 1, with 0 assigned under uncertainty; the prompt sets the final score to 0 if no actual edit is performed and gives greater importance to Instruction Following for knowledge-dependent tasks. The final score also considers important uncovered issues, with anchors at 0, 5, and 8.Source: System Prompt for the Image Editing… · pp. 30–31