PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoRubric-T2I: Robust Rule-Based Reward Model for Text-to-Image Alignment

Kuei-Chun Kao, Daixuan Huo, Yuanhao Ban, Cho-Jui Hsieh

arXiv:2605.17602v2cs.AIcs.CVcs.LG

TL;DR

T2I reward models are costly, opaque, and vulnerable to missing fine-grained prompt constraints, while manually or heuristically designed VLM rubrics may not track human preferences. AutoRubric-T2I learns sparse weighted rubrics from preference data to guide VLM judges, outperforming strong baselines on MMRB2 and improving downstream T2I generation with Flow-GRPO.

  • Problem

    Existing T2I reward models require large preference corpora and compress multidimensional judgments into opaque scores, while VLM judges' manually or heuristically specified criteria may misalign with human preferences.

  • Method

    AutoRubric-T2I learns a sparse weighted set of natural-language rubrics from preference data using VLM-scored rubric features, ℓ1-regularized selection, and hard-pair-driven refinement.

  • Results

    AutoRubric-T2I outperforms existing rubric-generation and competitive scalar reward-model baselines on MMRB2 and improves downstream T2I generation on TIIF and UniGenBench++ with Flow-GRPO.

  • Takeaways & Limitations

    Learned weighted rubrics provide a practical and interpretable alternative to scalar reward models for robust T2I alignment.

  • Takeaways & Limitations

    The reported formulation uses only a small preference sample, such as M = 256, so its evidence is bounded by that data setting.

Abstract

from arXiv · show

Aligning Text-to-Image (T2I) generation models with human preferences increasingly relies on image reward models that score or rank generated images according to prompt alignment and perceptual quality. Existing reward models are commonly trained as Bradley-Terry (BT) preference models on large-scale human preference corpora, making them costly to train, difficult to adapt, and opaque in their evaluation criteria. Meanwhile, Vision-Language Model (VLM) judges can provide more fine-grained assessments through textual rubrics, but their manually designed or heuristically generated scoring rules may fail to reliably reflect human preferences. In this paper, we propose AutoRubric-T2I, the first rubric learning framework in T2I that automatically synthesizes and selects explicit rubrics for guiding VLM judges. AutoRubric-T2I first synthesizes reasoning traces from preference pairs into candidate rubrics, then uses a VLM judge to score paired images under each rubric, producing pairwise rubric-score differences for preference learning. To remove noisy and redundant rules, we further employ a $\ell_1$-Regularized Logistic Regression Refiner, which selects the Top-$N$ most discriminative rubrics. Extensive evaluations show that AutoRubric-T2I produces high-quality, interpretable reward signals using less than 0.01% of the annotated preference data, substantially reducing the need for large-scale reward-model training. On image reward benchmarks such as MMRB2, AutoRubric-T2I outperforms strong reward model baselines. We further validate AutoRubric-T2I as an RL reward on downstream T2I tasks, including TIIF and UniGenBench++, where it improves generation quality over scalar reward models using the Flow-GRPO pipeline on diffusion models.

1 Introduction

Image reward models support T2I preference alignment but are costly and opaque, while VLM judges offer fine-grained assessment without reliably matching human preferences. AutoRubric-T2I learns explicit, interpretable rubrics to guide VLM-based reward modeling without fine-tuning.

  • Image reward models rank generated images against prompts and human quality expectations, supporting candidate selection, filtering, and reinforcement learning.
  • Large-scale Bradley-Terry preference models capture human preferences but require massive annotations, expensive fine-tuning, and compress multiple criteria into opaque scalar scores.
  • Prompted VLM judges assess fine-grained visual correctness but can perform up to 10% worse than learned BT models on HPS or PickScore preference datasets.
  • Existing rubric methods leave open how to automatically derive, select, and refine explicit criteria that better align with human preferences.
  • AutoRubric-T2I automatically derives and refines rubrics for off-the-shelf VLM judges, learning which criteria predict human preferences without training a dense scalar reward model.
  • The framework combines sparse logistic rubric learning, ℓ1 regularization, coordinate selection, and hard-pair mining to refine the rubric set.
  • Its contributions include compact weighted rubrics, failure-driven refinement, strong MMRB2 preference prediction, and improved TIIF and UniGenBench++ downstream alignment.

2 Related Work

Related work shows that scalar rewards can obscure prompt-specific errors and invite reward hacking, while rubric methods improve interpretability but often depend on manual or heuristic construction. AutoRubric-T2I instead learns a sparse global rubric set offline from image preference data.

  • Scalar reward models compress multidimensional preferences into implicit scores, limiting interpretability and enabling exploitation of superficial visual features.
  • Rubric-based evaluation makes preferences more interpretable through explicit criteria, but prior automatic rubric methods primarily reduce reliance on manually written rules.
  • AutoRubric-T2I learns a sparse, weighted global rubric set directly from image preference data and uses ℓ1-regularized logistic regression to select discriminative rubrics.
  • RubricRL constructs prompt-specific checklists during reinforcement learning, whereas AutoRubric-T2I learns a global rubric set offline from preference data.

3 Preliminaries

AutoRubric-T2I replaces opaque scalar reward modeling with weighted natural-language rubrics scored by a VLM and selected through sparse preference learning. Its iterative pipeline generates, prunes, and refines rubrics using small preference-pair data and hard failure cases.

  • 3 Preliminaries: Standard T2I reward models learn scalar scores or pairwise preferences from preference data, but scalar compression can enable reward hacking.HPSv3 optimization can achieve high reward while violating prompt constraints, whereas AutoRubric-T2I favors rubric-aligned generations.
  • 4 Preliminaries: AutoRubric-T2I represents each rubric as a natural-language prompt and obtains a continuous VLM judge score for each prompt-image pair.The score lies in [0, 1] and is the predicted probability of the yes token.
  • 4 Preliminaries: The final reward combines N rubric scores with learned weights chosen to best explain observed human preference pairs.The training data contains prompts, two generated images, and preference labels; the framework can operate with M = 256 pairs.
  • 4.1 Formulation: ℓ1-regularized logistic regression assigns zero weights to weak or redundant rubrics while retaining the most predictive criteria.The method relaxes top-N selection over an infinite rubric space with an ℓ1 penalty.
  • 4.2 Implementation: Iterative block coordinate descent adds candidate rubrics from current failure cases, then prunes zero-weight rubrics to maintain a compact working set.The practical pipeline scores training pairs, selects Top-N rubrics, evaluates them on validation data, mines hard pairs, and expands the rubric set.
  • 4.2 Implementation: The framework initializes rubrics from diversity-selected seed pairs, generates objective statements with VLM chain-of-thought prompting, and refines them through hard-pair mining.The default seed set contains 256 preference pairs selected for preference margin and prompt-level diversity.

5 Experiments

AutoRubric-T2I is evaluated on preference benchmarks and downstream T2I reinforcement learning, where it improves VLM-based preference prediction and generation outcomes over scalar and generic rubric rewards.

  • Experimental Setup: The evaluation compares AutoRubric-T2I with scalar reward models, fine-tuned VLM reward models, zero-shot VLM judges, and adapted rubric baselines.Experiments cover MMRB2, in-domain PickScore and HPSv3, and downstream TIIF and UniGenBench++ tasks.
  • Preference Benchmark Evaluation: 62.5% overall accuracy on MMRB2 is achieved by Qwen3-VL-8B with AutoRubric-T2I learned from HPSv3 preferences, exceeding direct pairwise evaluation and fine-tuned scalar baselines.The same judge reaches 62.4% when rubrics are learned from PickScore preferences; stronger judges reach 67.7% and 71.4%.
  • Preference Benchmark Evaluation: 63.2% on PickScore and 63.9% on HPSv3 are obtained by Qwen3-VL-8B with AutoRubric-T2I on in-domain test sets, while Gemini-3-Flash reaches 70.3% and 70.0%.Fine-tuned scalar models remain strong on distributions represented in their training data.
  • Downstream T2I Reinforcement Learning: AutoRubric-T2I uses a weighted sum of rubric-level yes-token probabilities as a dense reward while fine-tuning SD-3.5-Medium with Flow-GRPO.The reward uses Qwen3-VL-8B guided by the learned rubric set.
  • Downstream T2I Reinforcement Learning: 71.6% on short prompts and 67.9% on long prompts are reached on TIIF under HPSv3, improving SD-3.5-Medium from 65.3% and 62.7%, respectively.Under PickScore, the method reaches 70.8% and 69.0% overall and shows particularly strong gains in reasoning, relation composition, and text generation.
  • Downstream T2I Reinforcement Learning: 62.7% on short prompts and 66.9% on long prompts are reached on UniGenBench++ under HPSv3, improving the base model from 61.0% and 64.0%.PickScore-derived rubrics reach 62.4% and 67.7% overall, with stronger improvements in relation, compound reasoning, layout, and text rendering.

6 Discussion and Analysis

Ablations identify positive rubric weighting, hard-pair refinement, and cluster-based initialization as important components, while qualitative and human evaluations favor AutoRubric-T2I outputs.

  • Ablation Study: Deconstructing AutoRubric-T2I: Unrestricted ℓ1 refinement provides little improvement because negative rubric weights make the learned reward difficult to interpret.Positive-only weights improve performance by making rubric satisfaction consistently additive to the reward.
  • Ablation Study: Deconstructing AutoRubric-T2I: Hard-pair mining produces larger gains than randomly sampled refinement pairs because difficult preference pairs expose missing or ambiguous criteria.Multi-round refinement helps, but random sampling provides only modest gains.
  • Ablation Study: Deconstructing AutoRubric-T2I: 62.5% on MMRB2 under HPSv3 and 62.4% under PickScore are achieved with cluster-based initialization, up from 59.1% and 56.4%.The ablation indicates that both initial rubric coordinates and the subsequent hard-pair refinement trajectory matter.
  • Qualitative and Human Evaluation of the RL Policy: AutoRubric-T2I better preserves requested objects, relations, and scene structure than scalar-reward and generic-rubric optimization in qualitative comparisons.The authors present these observations as suggesting more targeted feedback during reinforcement learning.
  • Qualitative and Human Evaluation of the RL Policy: 44.8% selection rate is achieved in a 4-way human evaluation, above the 25% random baseline with p < 0.001.The evaluation used 30 annotators and 20 prompts, with a 95% confidence interval of [40.7%, 48.9%].

7 Conclusion

AutoRubric-T2I learns interpretable, weighted rubrics from preference data instead of training a dense scalar reward model. Its VLM-based reward is refined through sparse selection and hard-pair mining, and supports downstream Flow-GRPO optimization.

  • Conclusion: AutoRubric-T2I learns a compact weighted set of natural-language rubrics from image preference data without fine-tuning.The framework derives interpretable criteria using VLM-scored rubric features, sparse selection, and hard-pair-driven refinement.
  • Pipeline: The pipeline scores rubric features with a VLM, retains Top-N positive-weight rubrics through ℓ1-regularized logistic regression, and iteratively mines misranked hard pairs.Validation accuracy determines the best retained rubric set before further rubric generation.
  • Downstream optimization: The learned weighted rubric reward replaces the scalar reward in Flow-GRPO without modifying the upstream optimization algorithm.Rubric scores are combined using coefficients fit on 256 preference pairs, with no reward retuning during reinforcement learning.
  • Comparison with prior methods: AutoRubric-T2I makes rubric rewards reusable across evaluation and reinforcement fine-tuning by learning a compact global rubric set offline.This differs from prompt-adaptive methods that generate new rubrics for each prompt during reinforcement learning.
  • Comparison with prior methods: Unlike AutoRule and Auto-Rubric, AutoRubric-T2I explicitly learns preference-discriminative rubric weights rather than using uniform weighting, majority voting, or semantic diversity alone.RubricRL instead generates prompt-specific criteria and uniformly averages their scores.

E Dataset and Benchmark Details

AutoRubric-T2I is trained from small, independently sourced preference subsets and evaluated across held-out preference and generative T2I benchmarks. The benchmark suite spans both standard preference prediction and instruction-following or semantic-consistency evaluation.

  • Training preference corpora: The method uses HPDv3 and PickScore independently as source preference distributions.Each corpus supplies training and held-out validation data for rubric learning.
  • Training preference corpora: Only 256 preference pairs per source corpus are used for seed-rubric generation, with an additional held-out split for validation.Validation monitors the best Top-N rubric set across refinement rounds.
  • Seed selection: The 256 seed pairs combine proxy-reward preference margins with clustered prompt diversity rather than uniform sampling.This selection follows a FiFA-inspired two-factor strategy.
  • Preference benchmarks: Held-out preference evaluation covers official HPDv3 and PickScore test splits plus MMRB2’s four multimodal preference subtasks.MMRB2 contains 1,000 expert-annotated pairs per subtask drawn from 23 frontier models across 21 source tasks.
  • Generative T2I benchmarks: Downstream generative evaluation uses TIIF for instruction fidelity and UniGenBench++ for semantic consistency across long and short prompts.Both benchmarks assess T2I quality after reinforcement-learning post-training.

F Details of Hyperparameter for AutoRubric-T2I

The reported AutoRubric-T2I configuration uses a fixed sparse-selection and hard-pair-refinement schedule shared across source corpora. Qualitative evaluation examines prompt faithfulness, compositional accuracy, and visual quality after Flow-GRPO fine-tuning.

  • Refinement configuration: The final pipeline runs 10 refinement rounds on 256 preference pairs and retains Top-20 rubrics.Each round mines 16 hard pairs, with stale pairs discarded after being selected more than four times.
  • Sparse rubric selection: Sparse logistic regression uses an ℓ1 penalty, C=1.0, no intercept, random state 42, and a 10^-4 coefficient threshold.Positive coefficients are sorted and the top 20 define the retained rubric set.
  • Hard-pair mining: Hard-pair mining partitions misranked examples into small-margin, large-margin, and high-reward-wrong buckets using margin percentile 0.3 and reward quantile 0.7.Sixteen pairs are sampled each round with phase-dependent bucket weights.
  • Hard-pair mining: Early refinement emphasizes large-margin errors, whereas late refinement shifts toward high-reward-wrong cases for finer-grained rubric generation.The schedule changes from (0.6, 0.4, 0.0) early to (0.3, 0.3, 0.4) late.
  • Qualitative evaluation: Qualitative examples compare Flow-GRPO models using AutoRubric-T2I rewards on UniGenBench++ and TIIF.The examples illustrate prompt faithfulness, compositional accuracy, and visual quality relative to the base model.

H Runtime and Data-needed Analysis

AutoRubric-T2I shifts reward-modeling costs from large-scale annotation and gradient-based training to one-time rubric refinement. This reduces data requirements but increases per-image inference cost during reinforcement learning.

  • Data and training cost: 256 preference pairs replace the 137K–798K pairs required by CLIP-based reward models, with no neural reward-model training.The remaining major cost is one-time rubric refinement.
  • One-time refinement cost: The final refinement stage runs for 10 rounds and finishes in approximately 2–4 hours on 4× A6000 GPUs.The resulting weighted rubric set can be reused for downstream evaluation and reinforcement learning.
  • RL inference cost: At deployment, each generated image requires 20 VLM forward passes, one for each selected rubric.For 10K prompts with four rollouts each, this produces 800K rubric-scoring calls.
  • Overall trade-off: AutoRubric-T2I trades higher inference cost for lower annotation cost, no gradient-based reward-model training, and interpretable per-rubric scores.The trade-off supports fine-grained diagnostics and rubric-level reward shaping.

I Training Dynamics

AutoRubric-T2I shows steadily improving training and evaluation rewards with lower variance, while qualitative results indicate better preservation of prompt-specific content than scalar reward optimization.

  • AutoRubric-T2I steadily improves training and evaluation rewards while maintaining substantially lower reward variance.HPSv3 shows larger reward dispersion, whereas PickScore has extremely low variance.
  • Scalar reward optimization can increase reward while visual quality degrades notably.
  • AutoRubric-T2I better preserves prompt-specific objects, relations, and fine-grained details than the base model and alternative reward methods.

J Limitations and Broader Impact

The paper identifies domain-specific rubric weights as a limitation and highlights both rubric-aware RL integration and inspectable, auditable optimization criteria as broader implications.

  • Limitations: Rubric weights fit to HPSv3 or PickScore preferences may shift on out-of-domain prompts such as typography-heavy or highly stylized images.The paper suggests refitting weights on 256 in-domain pairs or learning prompt-conditional weights.
  • Limitations: AutoRubric-T2I can serve as a drop-in Flow-GRPO reward through a weighted sum of per-rubric scores.Rubric-level rewards may support finer-grained credit assignment, but combining them with scalar rewards for advantage estimation remains future work.
  • Broader Impact: Human-readable rubrics and learned weights make T2I optimization criteria inspectable and auditable, helping diagnose and correct biases through the weights.

L Quantitative Human Evaluation

The human evaluation uses a randomized forced-choice survey in which annotators select the best image among four T2I policies according to prompt alignment and visual quality.

  • Evaluation Setup: 600 human judgments were collected from 30 graduate-student annotators evaluating 20 T2I prompts each.
  • Evaluation Setup: Each question presents four anonymized candidates from the base model, scalar-reward optimization, AutoRule-based optimization, and AutoRubric-T2I.
  • Evaluation Setup: Annotators select exactly one image that best satisfies the text prompt, considering visual quality and prompt alignment.The display order is randomized to reduce positional bias, avoiding calibrated numerical scoring.
  • Rubric Analysis: Final rubric sets rank rules by fitted ℓ1-regularized logistic-regression weights, pruning rules with effectively zero weight from the displayed Top-N set.
Loading 2605.17602v2…