RubricRL: Simple Generalizable Rewards for Text-to-Image Generation

1 Introduction

• Existing reward mixtures require manually tuned weights, while unified scalar rewards can obscure reasoning and limit extensibility and developer control.The introduction describes these as the two broad categories of current text-to-image RL rewards.Source: §1 · p. 2
• RubricRL builds a prompt-specific checklist of fine-grained visual criteria, scores each independently with a multimodal judge, and weights relevant dimensions adaptively.Its signals integrate with policy optimization frameworks such as GRPO, PPO, or DiffusionNFT.Source: §1 · p. 2
• The decomposable reward lets developers inspect, extend, and adjust rewarded or penalized aspects, with per-criterion diagnostics for evaluation and debugging.The paper characterizes this process as auditable rather than a black-box heuristic.Source: §1 · p. 2
• RubricRL is architecture-agnostic across diffusion and autoregressive models, and its prompt-adaptive rewards reflect salient prompt details without manual tuning.Examples include numerals, named entities, styles, and embedded text; rubric outputs also support variance-aware group advantages for long-horizon rollouts.Source: §1 · p. 3
• RubricRL improves prompt faithfulness, compositional accuracy, and visual realism while maintaining generalizability across architectures and datasets.The authors also report more consistent optimization behavior and controllable, interpretable reward shaping.Source: §1 · p. 3

2.1 Text-to-Image Generation Methods.

• Diffusion and autoregressive models provide distinct routes from text to images, and the paper presents its reward design for both model families.Autoregressive systems represent images as discrete token sequences and model text and image tokens with a transformer backbone.Source: §2.1 · pp. 3–4

2.2 Reinforcement Learning for Text-to-Image Generation.

• Likelihood training can under-optimize user-salient qualities, while RL offers task-aligned feedback for properties such as semantic faithfulness, compositional accuracy, and aesthetics.The related work describes RL applications to diffusion and unified autoregressive text-to-image models.Source: §2.2 · p. 4
• Rubric-based rewards replace tuned mixtures and scalar preference models with prompt-specific criteria scored independently and transparently aggregated.The paper identifies text alignment, OCR, object count, spatial relations, and overall coherence or quality as example criteria, and claims to be the first to propose rubric-based rewards in text-to-image RL.Source: §2.2 · pp. 4–5

3 Method

• RubricRL shares rubric construction and rubric-guided rewards across autoregressive and diffusion backbones, using GRPO for AR and DiffusionNFT for diffusion.Source: §3 · p. 5

3.1 Overall Architecture

• Prompt-specific rubrics replace multiple specialized graders with interpretable criteria, addressing their computational cost and calibration demands.Source: §3.1 · p. 6

3.2 Rubric-based Reward

• GPT-o4-mini generates candidate criteria, selects the ten most relevant, scores each image against them, and averages the binary judgments into a normalized reward.Source: §3.2 · pp. 6–7

3.3 Reinforcement Learning with GRPO

• GRPO standardizes each rollout’s reward relative to its group and applies a clipped PPO-like objective, stabilizing training across heterogeneous reward scales and noisy evaluations.Source: §3.3 · p. 7

3.4 Dynamic Rollout Sampling

• Dynamic sampling oversamples per prompt, then selects the top-K high-reward rollouts and randomly samples the rest; the authors report better stability, diversity, and model quality than original GRPO and DAPO.Source: §3.4 · p. 8

4.1 Implementation Details

• The implementation trains on 11,000 images and evaluates Phi3-3.8B and Qwen2.5-0.5B AR backbones, with additional fine-tuning of Qwen-Image and FLUX diffusion models.The AR backbones start from SFT checkpoints; the diffusion models use DiffusionNFT.Source: §4.1 · pp. 8–9

4.2 Comparing with State-of-the-arts

• RubricRL surpasses X-Omni by approximately 4% on GenEval with both AR backbones, and improves over Qwen-Image by +6.85% and FLUX by +8.48% on diffusion benchmarks.On diffusion DPG-Bench, the gains are smaller but consistent.Source: Table 1 · p. 9, Table 2 · p. 9, §4.2 · pp. 10–11

4.3 Ablation Study

• The hybrid rollout sampler performs best among the compared strategies by balancing high-reward rollouts with diverse candidates.FFKC-1D and DAPO do not outperform the vanilla baseline in the reported comparison.Source: §4.3 · pp. 11–12, Table 5 · p. 11
• Normalizing GRPO advantages over retained rollouts performs better than normalizing over all candidates, because it better reflects the reward distribution guiding learning.The authors say this avoids high-variance or low-quality rollouts distorting the gradient direction.Source: Table 6 · p. 11, §4.3 · p. 12
• RubricRL exceeds SFT Best-of-N sampling (N = 8) by over 5% on GenEval, although Best-of-N sampling also achieves significantly higher scores.The comparison samples eight SFT rollouts per prompt, scores them with rubrics, and selects the top four.Source: §4.3 · p. 12, Table 7 · p. 12
• GPT-o4-mini can misjudge object counts in ambiguous, poor-quality images, and the issue is less pronounced for higher-quality generations.The authors link this failure to weaker counting-related results for Qwen2.5-0.5B; it almost disappears with Phi3-3.8B.Source: §4.3 · pp. 12–13, Figure 4 · p. 13

4.4 Visual results

• Qualitative comparisons show RubricRL images as more aesthetically appealing and semantically aligned, including better handling of fine-grained instructions than the illustrated baselines.The examples include baseline outputs that mishandle a black bag or omit a wooden cane.Source: §4.4 · pp. 13–14, Figure 6 · p. 14

• RubricRL provides prompt-adaptive, decomposable supervision through configurable visual criteria that are scored independently.The paper names counting, attributes, OCR fidelity, and realism as examples.Source: §5 · p. 14
• The rubric-based reward framework produces interpretable and modular signals that integrate with standard RL policy optimization.Source: §5 · p. 14
• Experiments show RubricRL outperforms existing RL-based approaches for enhancing text-to-image generation.Source: §5 · p. 14

• The supplement is titled “Generalizable Rewards for Text-to-Image.”Source: Supplementary Material of “RubricRL:… · p. 19
• The supplementary material covers additional methodology details, including differences between autoregressive and diffusion models and DiffusionNFT.Source: Supplementary Material of “RubricRL:… · p. 19
• It also presents additional ablation and visualization results to further demonstrate the method's effectiveness.Source: Supplementary Material of “RubricRL:… · p. 19

A.1 Prompt Templates for Rubric Generation and Evaluation

• The prompting pipeline generates candidate rubrics from the prompt, selects the most relevant when the set is large, and scores each image against each selected rubric with a binary value.The three stages determine what is evaluated, keep the reward compact, and test whether each explicit visual requirement is satisfied.Source: §A.1 · p. 19, §A.2 · p. 19, Figure 1 · p. 20

A.2 Prompt Templates for Rubric Generation and Evaluation

• The prompt templates are provided to improve reproducibility of RubricRL's prompting procedure.Source: §A.1 · p. 19, §A.2 · p. 19

A.3 Difference between AR and Diffusion-based Models

• AR generators sample discrete visual-token sequences using next-token prediction, whereas diffusion and flow-matching generators iteratively transport continuous latents before decoding pixels.AR codes are reshaped into a latent grid and decoded; diffusion sampling integrates a learned time-dependent velocity field.Source: §A.3 · pp. 19–21

A.4 Reinforcement Learning with DiffusionNFT

• DiffusionNFT integrates rubric rewards into diffusion policy optimization using velocity-space updates and within-prompt group-relative reward normalization.Rubric rewards are converted into bounded optimality probabilities using normalized group rewards; the method is described as compatible with diffusion velocities.Source: §A.4 · pp. 22–23

B.1 Continuous vs. Binary Reward Supervision

• 84.68% with binary rewards versus 81.70% with continuous rewards on GenEval; binary supervision helps discrete correctness criteria, while finer scores provide denser feedback for nuanced attributes.The passage highlights object counting and multi-object color verification as discrete criteria, and color as a perceptually nuanced example.Source: §B.1 · pp. 23–24

B.2 Ablation on the Number of Rubrics M

• M = 10 achieves the best overall performance; larger rubric sets improve results through M = 10, while M = 12 scores lower overall despite a slight Counting gain.The authors attribute the decline at larger sizes to possible redundant or weakly relevant rubrics increasing training noise.Source: §B.2 · p. 24

B.3 Analysis of Using Different Models as Graders

• Accurate judgments on criteria such as counting, spatial relations, and color make rewards more informative for policy updates; weak or noisy graders can hurt stability and sample efficiency.The passage says policies may overfit or exploit misaligned signals from weak graders.Source: §B.3 · p. 24
• Qwen2.5-VL 32B outperforms its 3B and 7B variants, but all three lag behind o4-mini on GenEval.The authors associate o4-mini’s advantage with stronger instruction following, multi-step reasoning, and alignment with the rubric design.Source: §B.3 · p. 25, Table 3 · p. 25

B.4 Analysis of the Number of Rollouts Before and After Dynamic Sampling

• The procedure generates N′ candidate rollouts per prompt, selects N with Hybrid dynamic sampling, and uses those selected samples to compute the GRPO loss.The candidates are described as N′ > N.Source: §B.4 · pp. 25, 27
• A 4× oversampling ratio achieves comparable overall performance; beyond an initial benefit, larger budgets can add reward variance and noisier advantage estimates.Selecting too many rollouts can also include low-quality generations and dilute the learning signal.Source: §B.4 · pp. 27–28

B.5 Evaluation on Aesthetic Quality

• For Phi3, RubricRL raises the aesthetic score from 4.9621 to 5.3562 and achieves the best result among the compared reward-learning baselines.The comparisons include CLIPScore, HPSv2, Unified Reward, LLaVA-Reward-Phi, AR-GRPO, and X-Omni.Source: §B.5 · p. 28
• For Qwen-Image, RubricRL raises the aesthetic score from 5.3297 to 5.6801, achieving the second-best result and remaining competitive with HPSv2.HPSv2 is described as explicitly designed to capture human preference and aesthetic quality.Source: §B.5 · p. 28
• The authors conclude that RubricRL’s gains are not limited to compositional metrics and can extend to broader visual quality aspects.The aesthetic evaluation uses a pretrained CLIP+MLP reward model independent of the rubric rewards used in training.Source: §B.5 · p. 28

C User study Criteria and Interface

• 98.1% of generated rubrics were judged prompt-consistent, 0.95% contained hallucinated constraints, and GPT-o4-mini agreed with human judgments in 94.8% of evaluated cases.The study sampled 100 prompts, generated 10 rubrics per prompt, and evaluated 1,000 prompt–rubric pairs.Source: C User study Criteria and Interface · pp. 29–30
• Human preference studies used anonymized, randomized images and assessed prompt following and perceptual quality through top-rank and pairwise comparisons.The pairwise protocol included an “Equal” option for comparable images.Source: C User study Criteria and Interface · pp. 30–31

D Cost, latency, and accessibility.

• RubricRL’s o4-mini API calls cost about $1.18K per epoch; on Phi3, it reached 0.8228 on GenEval after 1 epoch, versus AR-GRPO’s 0.8001 after 3 epochs.The passage notes that general-purpose rubric grading is challenging for smaller models because criteria can be diverse, abstract, and image-dependent.Source: D Cost, latency, and accessibility · p. 32

E Visualization

• Additional visualizations show RubricRL generations for complex prompts and display each prompt’s key-criterion rubrics alongside rollout correctness by criterion.The text identifies these as Figures 4 and 5, respectively.Source: E Visualization · pp. 32–33