Source-linked AI summary
RubricRM: Generative Reward Modeling via Dynamic Rubrics for Image Generation and Editing
Zijian Kan, Wei Wang, Long Luo, Bing Zhao, Xuan Ren, Weixu Qiao, Wenbo Li, Hu Wei, Lin Qu
TL;DR
Existing visual reward models often use scalar scores or fixed criteria, limiting instruction-adaptive and inspectable evaluation. RubricRM generates an input-specific rubric and applies weighted dimension-level scoring, using SFT followed by GRPO. Across generation and editing benchmarks, it outperforms specialized reward-model baselines while preserving an inspectable judgment trace.
Problem
Existing visual reward models often use scalar scores or fixed evaluation dimensions, although different instructions require distinct assessment criteria.
Method
RubricRM generates an input-specific rubric with task intent, dimensions, weights, and scoring criteria, then scores candidate images using a two-stage SFT and dimension-level GRPO pipeline.
Results
RubricRM outperforms specialized reward-model baselines across text-to-image and image-editing benchmarks; RubricRM-Gen-9B reaches 72.00 on MMRB2.
Takeaways & Limitations
Dynamic rubric generation makes the scoring protocol explicit and preserves an inspectable judgment trace across generation and editing evaluation.
Takeaways & Limitations
Rubric trace synthesis relies on a proprietary teacher model, and the framework currently evaluates static images rather than video or temporal editing.
Abstract
from arXiv · showhide
Reward models play an essential role in aligning visual generative models, yet most existing visual reward models use a single scalar score or rely on fixed criteria that cannot adapt to different instructions. This limits both interpretability and task sensitivity, especially for text-to-image generation and instruction-based image editing, where different inputs require different evaluation dimensions. We propose RubricRM, a pairwise generative reward modeling framework that first produces an input-specific rubric with evaluation dimensions, weights, and scoring criteria, and then applies the rubric to score candidate images. We train dedicated RubricRM models for text-to-image generation and image editing using a two-stage training pipeline: supervised fine-tuning teaches the model the rubric-based scoring paradigm, while GRPO further improves scoring through fine-grained dimension-level rewards. Experiments on multiple generation and editing benchmarks show that RubricRM outperforms existing specialized reward models and remains competitive with strong proprietary MLLM judges despite using smaller backbones. Our models, data, and code are available at https://github.com/zijiankan/RubricRM.
1 Introduction
Existing visual reward models often provide scalar or fixed-criterion evaluations that do not adapt to instruction-specific needs. RubricRM addresses this with input-specific rubrics, rubric-grounded scoring, and two-stage training for text-to-image and image-editing preference selection.
- Motivation: Fixed evaluation dimensions cannot reflect that realistic and creative prompts prioritize different assessment criteria.Existing scalar rewards also provide limited interpretability and fine-grained semantic sensitivity.
- Approach: RubricRM generates an input-specific rubric specifying evaluation dimensions, weights, and scoring criteria before aggregating weighted per-dimension scores into a preference.The rubric functions as an inspectable intermediate protocol between the instruction and preference decision.
- Training: Training combines rubric-trajectory supervised fine-tuning with dimension-level GRPO to learn rubric formulation and refine scoring accuracy.SFT teaches decomposition into intent, dimensions, weights, and criteria, while reinforcement learning aligns dimension-level score differences with reference differences.
- Approach: RubricRM supports preference selection for both text-to-image generation and image editing within a single inference pass.At inference, it first generates the rubric and then scores each candidate image along its dimensions.
- Results: 72.00, 74.12, and 84.45 are RubricRM-Gen-9B accuracies on MMRB2, GenAI-Bench, and GenAI-Bench-Verified, respectively.RubricRM-Gen-9B achieves the highest scores among reward-model baselines on all three text-to-image benchmarks.
- Results: RubricRM obtains strong results across multiple reward benchmarks, outperforming specialized reward-model baselines on generation and editing tasks.The models are trained using multiple open-source preference datasets with taxonomy annotation, balanced sampling, and supplementary expert annotations.
2 Related Work
Prior visual evaluation methods range from scalar preference rewards to generative judges and rubric-based systems, but many use fixed or externally supplied criteria. RubricRM instead trains a self-contained model to generate input-specific dimensions and weights for both generation and editing evaluation.
- Text-to-image evaluation: CLIP-based metrics are efficient but have limited sensitivity to fine-grained compositional semantics.Preference-trained alternatives improve human alignment while commonly retaining scalar reward outputs with limited interpretability.
- Text-to-image evaluation: Generative reward models can provide evaluative reasoning, yet typically rely on predefined fixed evaluation dimensions.Context-adaptive assessment does not necessarily specify each criterion’s detailed requirements or assign criterion weights.
- Image editing evaluation: Image-editing rewards must balance instruction satisfaction with preservation of irrelevant source content.Existing editing models include scalar preference rewards and generative evaluators that do not define instruction-specific rubrics.
- Rubric-based evaluation: Recent rubric systems make criteria explicit through task-adapted rubrics, rubric resources, synthetic rules, or prompt-conditioned binary criteria.Their scoring may rely on external frozen vision-language models or globally shared rules.
- Rubric-based evaluation: RubricRM differs by training a standalone, self-contained generative reward model that produces input-specific dimensions and normalized weights.It applies these generated criteria directly to text-to-image and image-editing preference evaluation.
3 RubricRM
RubricRM defines rubric-based visual reward modeling for text-to-image generation and image editing, producing interpretable, instruction-specific criteria and preference decisions. Its data construction and two-stage training jointly target broad instruction coverage, rubric generation, and dimension-level scoring accuracy.
- 3.1 Task Definition: RubricRM evaluates text-to-image prompts or editing instructions with two candidate images and predicts which better satisfies the instruction.For editing, the input additionally includes a source image and editing instruction.
- 3.1 Task Definition: The model generates atomic dimensions, normalized weights, graded criteria, and per-candidate scores, then selects the image with the higher weighted total.The rubric specifies what to evaluate, each aspect’s importance, and the meaning of score levels.
- 3.2 Rubric Trajectory Data: RubricRM uses classification-driven balanced sampling to cover diverse instruction types and underrepresented visual scenarios.Samples may receive multiple labels, so multi-label stratified sampling balances coverage across second-level categories.
- 3.2 Rubric Trajectory Data: Training data combine open-source text-to-image preferences, supplementary annotated prompts, and 30,695 image-editing samples.Supplementary text-to-image data target underrepresented categories, while editing data are built from EditReward-Data without additional prompt synthesis.
- 3.2 Rubric Trajectory Data: The data analysis identifies long-tail categories such as text rendering (6.45%), logical reasoning (8.98%), and world knowledge (15.83%) in text-to-image generation.Balanced sampling is designed to retain coverage of these less prevalent categories.
- 3.2 Rubric Trajectory Data: Rubric trajectories are teacher-synthesized from human preference labels and contain intent, weighted dimensions, graded descriptors, and candidate assessments.Quality control removes malformed trajectories and checks that weighted scores produce the labeled preference direction.
- 3.3 Two-Stage Training: Dedicated Qwen3.5-4B and Qwen3.5-9B backbones follow the same two-stage training methodology for the two tasks.Supervised fine-tuning teaches rubric-based scoring, followed by reinforcement learning with fine-grained dimension-level rewards.
- 3.3 Two-Stage Training: GRPO rewards dimension-level score-gap agreement, leaving matching directions unpenalized, discounting tie mismatches, and heavily penalizing reversals.The reward uses ϕi values of 1.0, 0.6, and 0.1 for matching signs, one zero gap, and opposite nonzero signs, respectively.
4 Experiments
RubricRM is evaluated on pairwise text-to-image and image-editing benchmarks against general MLLM judges and specialized reward models. It achieves strong reward-model performance, with SFT contributing most gains and GRPO adding consistent improvements.
- Experimental Setup: The experiments cover pairwise preference benchmarks for text-to-image generation and image editing, alongside general MLLM and specialized reward-model baselines.Text-to-image evaluation uses MMRB2, GenAI-Bench, and GenAI-Bench-Verified; editing uses MMRB2, EditReward-ERB, and EditScore-ERB.
- Text-to-Image Results: 72.00, 74.12, and 84.45 are RubricRM-Gen-9B’s scores on MMRB2, GenAI-Bench, and GenAI-Bench-Verified, respectively.These are the highest scores among reward-model baselines on all three text-to-image benchmarks.
- Image-Editing Results: 75.40, 46.40, and 85.64 are RubricRM-Edit-9B’s scores on MMRB2, EditReward-ERB Avg, and EditScore-ERB Avg, respectively.RubricRM-Edit-9B leads reward-model baselines on these benchmark-level scores, while RubricRM-Edit-4B obtains the highest EditReward-ERB K=3 score.
- Image-Editing Results: RubricRM obtains high scores across instruction following, source-image consistency, and visual quality in fine-grained image-editing evaluation.Its results remain competitive across subject, appearance, scene, and advanced editing categories.
- Ablation Study: Rubric-trajectory SFT produces most improvement, while dimension-level GRPO yields smaller but consistent additional gains across all benchmarks.SFT gains range from 6.8 to 13.1 points for 4B and 3.4 to 10.7 points for 9B on text-to-image benchmarks; editing gains range from 6.5 to 11.1 and 7.6 to 13.1 points, respectively.
5 Conclusion
RubricRM makes visual evaluation explicit by generating dynamic rubrics and then applying weighted, per-dimension scores. Across generation and editing benchmarks, it improves over prior specialized reward models while preserving an inspectable judgment trace.
- Conclusion: RubricRM represents visual evaluation as dynamic rubric generation followed by rubric-grounded scoring.The model first determines what should be evaluated, then applies those criteria through weighted per-dimension scores.
- Conclusion: A two-stage training procedure uses SFT for rubric formulation and GRPO for scoring alignment.This separates learning the evaluation protocol from aligning score application.
- Conclusion: RubricRM improves over prior specialized reward models across generation and editing benchmarks while preserving an inspectable judgment trace.The explicit scoring protocol supports traceable visual evaluation.
Limitations
RubricRM’s limitations concern teacher dependence and scope: synthesized rubric traces rely on a proprietary teacher model, and the framework currently evaluates static images only.
- Limitations: Rubric trace synthesis relies on a proprietary teacher model, so teacher biases may be inherited by RubricRM’s student models.The authors identify teacher-free rubric improvement and human-in-the-loop rubric correction as future directions.
- Limitations: The current framework does not address video generation or temporal editing, where temporal consistency would require additional dimensions and scoring mechanisms.The paper identifies video assessment as a future extension.
Ethical Considerations
RubricRM’s evaluative signals may inherit biases from human preference labels and teacher-synthesized rubric trajectories. Its rubrics should therefore be audited on target distributions and treated as model judgments rather than objective assessments.
- Human preference labels and teacher-synthesized rubric trajectories may introduce subjective or systematic biases into learned rubrics and preference judgments.
- Explicit dimensions, weights, and scores improve inspectability but are not objective standards or guaranteed causal explanations.
- Auditing on target content distributions and content-specific review are recommended before downstream deployment.
- Biases may be amplified when RubricRM is used to optimize text-to-image and image-editing models.
A.1 Training Data and Taxonomy Statistics
The training data uses stratified, multi-label taxonomy analysis to preserve broad coverage across text-to-image and image-editing categories. The appendix also explains how rubric dimensions and GRPO training are conditioned for reliable comparison.
- Training-data construction: Stratified sampling keeps maximum secondary-label coverage drift below 0.5 percentage points for both tasks.
- Taxonomy coverage: Text-to-image coverage is concentrated in Common Objects (92.4%), Style (69.0%), and Composition & Perspective (52.1%), while long-tail categories remain represented.
- Taxonomy coverage: Image-editing coverage is led by Object Editing (45.9%), Attribute Editing (25.7%), and Background Editing (18.2%), with additional editing categories represented.
- Taxonomy coverage: Common Objects+Style is the strongest text-to-image co-occurrence at 63.6%, motivating instruction-specific rubric dimensions for prompts combining subject, style, and composition.
- Rubric dimensions: Teacher-generated text-to-image rubrics emphasize subject fidelity, visual quality, scene composition, style, and prompt alignment.
- GRPO conditioning: Dimension-level rewards require rollout and reference assessments to share one teacher-generated rubric, while inference generates rubrics end to end.
A.4 Dimension-Level Reward Case Studies
The case studies show how dimension-level rewards distinguish correct preferences from correct reasoning, tie handling, and score-margin accuracy. These signals provide training feedback beyond the final pairwise label.
- Reward behavior: Dimension-level rewards penalize reversed directions most strongly, partially discount tie mismatches, and reward accurate score-difference magnitudes.
- Reversed dimension direction: A rollout can select the correct final preference while reversing a dimension direction, which final-preference-only rewards would miss.
- Tie mismatch: A tie mismatch applies λtie = 0.6 and lowers the aggregate reward from 0.950 to 0.760 despite the correct final preference.
- Score-margin accuracy: A rollout may match every dimension direction yet underestimate a reference margin, such as predicting 1/0 instead of 4/0.
B.1 Supplementary Results
Supplementary results indicate that rubric-based supervision contributes beyond preference-label exposure, while saturated-group filtering stabilizes continuous dimension-level GRPO training. Detailed editing results show competitive performance across multiple task categories.
- Detailed editing results: RubricRM remains competitive across subject, appearance, scene, and advanced editing categories rather than improving only one aspect.
- Label-only ablation: Label-only SFT improves over base models but remains consistently below rubric-based SFT on most benchmarks.
- GRPO stability: Without saturated-group filtering, late training shows reward collapse, response-length growth, and entropy collapse; filtering keeps these signals stable.
- Label-only ablation: For text-to-image generation, label-only SFT trails rubric-based SFT by 3.3–5.1 points with 4B and 1.9–3.9 points with 9B backbones.
- GRPO stability: Filtering groups with low reward standard deviation stabilizes policy updates by removing low-discrimination rollout groups that amplify numerical fluctuations.
B.2 Analysis of Generated Rubrics
RubricRM generates rubrics that are largely stable for repeated sampling while adapting their dimensions and weights to instruction type. The examples show how these rubrics translate task requirements into weighted, dimension-level image comparisons.
- B.2 Analysis of Generated Rubrics: Rubric analysis samples five stochastic rubrics per instance and compares matched dimensions by semantic similarity and weight difference.Higher dimension similarity indicates more consistent criteria, while lower weight difference indicates more consistent importance.
- B.2 Analysis of Generated Rubrics: 0.763 and 0.800 dimension similarities show stable repeated rubrics for generation and editing, respectively.Weight differences remain small at 0.102 for generation and 0.110 for editing.
- B.2 Analysis of Generated Rubrics: Rubric similarity follows same instruction > same type > different type for both generation and editing.For generation, same-type and different-type similarities are 0.597 and 0.563; editing reports 0.651 for same-type similarity before the passage truncates.
- B.3 Sensitivity to Teacher Choice: Human preference labels remain fixed while the teacher converts them into rubric trajectories, so teacher bias affects dimension selection and granularity rather than underlying labels.The cross-teacher analysis holds inputs and human preferences constant across three teacher families.
- B.3 Sensitivity to Teacher Choice: 91.5% of cross-teacher comparisons share at least one semantic dimension, 62.0% cover at least half of the smaller rubric, and average overlap is 0.534.Complete agreement is neither observed nor expected because different valid criteria can support the same human preference.
- D.1 Text-to-Image Generation Input Prompt: For the cranberry-bog example, the rubric weights prompt adherence and visual realism at 30% each, composition at 20%, and logical consistency at 20%.Image A scores 4.00 overall versus Image B's 1.10, with B penalized for repetitive textures, distorted geese, and illogical geometry.
D.2 Image Editing Editing Instruction
The image-editing example evaluates an educational airplane poster through text placement, thematic aesthetics, and preservation of the source object. Image A wins because it combines correct instructions with a more cohesive design and cleaner source preservation.
- D.2 Image Editing Editing Instruction: The editing instruction transforms a toy airplane image into a colorful educational kindergarten poster with specified text and placements.The required text appears at the top, left, right, and bottom, with the bottom message bolded.
- D.2 Image Editing Editing Instruction: Text Accuracy & Placement receives the largest weight at 40% and checks presence, spelling, exact locations, and bottom-text emphasis.The remaining dimensions are Thematic Style & Aesthetics at 30% and Content Preservation at 30%.
- D.2 Image Editing Editing Instruction: Image B satisfies all four text requirements and scores 4/4 for text accuracy and placement.The passage explicitly confirms correct spelling, requested locations, and bold bottom text.
- D.2 Image Editing Editing Instruction: Image A scores 4/4 for thematic style, while Image B scores 3/4 because its design feels more generic and less cohesive.Image A is described as bright, playful, and appropriate for a kindergarten poster.
- D.2 Image Editing Editing Instruction: Image A preserves the original airplane and adds non-distracting thematic elements, whereas Image B adds incorrect labels that detract from image integrity.This dimension evaluates preservation of the source centerpiece without unwanted artifacts or changes to its core identity.
- D.2 Image Editing Editing Instruction: The weighted totals are 4.00 for Image A and 3.10 for Image B.Both images follow the strict text and placement instructions, but the conclusion identifies Image A as superior overall.