PaperlayerView exact revision on arXiv

Source-linked AI summary

Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteria

Juanxi Tian, Fengyuan Liu, Jiaming Han, Yilei Jiang, Yongliang Wu, Yesheng Liu, Haodong Li, Furong Xu, Wanhua Li

arXiv:2605.08354v1cs.AI

TL;DR

Multimodal preference alignment often compresses multidimensional human judgments into opaque scalar or pairwise rewards. ARR externalizes preferences as explicit rubrics, while RPO uses them for training, improving evaluation and generation across text-to-image and image-editing benchmarks.

  • Problem

    Existing alignment methods collapse multidimensional human judgments into opaque scalar or pairwise signals, leaving reliable, interpretable, and data-efficient rubric generation unresolved.

  • Method

    ARR generates instance-conditioned, verifiable rubrics, and RPO uses rubric-conditioned binary preference decisions for policy optimization.

  • Results

    ARR-RPO improves evaluation and generation across text-to-image and image-editing benchmarks, including GenEval: 0.66 to 0.80.

  • Takeaways & Limitations

    Explicit rubrics provide a stable, factorized interface for more reliable and data-efficient multimodal preference alignment.

  • Takeaways & Limitations

    Fully automated rubric generation may lack the precision or domain-specific nuance achievable through curated human guidance.

Abstract

from arXiv · show

Aligning multimodal generative models with human preferences demands reward signals that respect the compositional, multi-dimensional structure of human judgment. Prevailing RLHF approaches reduce this structure to scalar or pairwise labels, collapsing nuanced preferences into opaque parametric proxies and exposing vulnerabilities to reward hacking. While recent Rubrics-as-Reward (RaR) methods attempt to recover this structure through explicit criteria, generating rubrics that are simultaneously reliable, scalable, and data-efficient remains an open problem. We introduce Auto-Rubric as Reward (ARR), a framework that reframes reward modeling from implicit weight optimization to explicit, criteria-based decomposition. Before any pairwise comparison, ARR externalizes a VLM's internalized preference knowledge as prompt-specific rubrics, translating holistic intent into independently verifiable quality dimensions. This conversion of implicit preference structure into inspectable, interpretable constraints substantially suppresses evaluation biases including positional bias, enabling both zero-shot deployment and few-shot conditioning on minimal supervision. To extend these gains into generative training, we propose Rubric Policy Optimization (RPO), which distills ARR's structured multi-dimensional evaluation into a robust binary reward, replacing opaque scalar regression with rubric-conditioned preference decisions that stabilize policy gradients. On text-to-image generation and image editing benchmarks, ARR-RPO outperforms pairwise reward models and VLM judges, demonstrating that explicitly externalizing implicit preference knowledge into structured rubrics achieves more reliable, data-efficient multimodal alignment, revealing that the bottleneck is the absence of a factorized interface, not a deficit of knowledge.

1 Introduction

ARR reframes multimodal alignment as explicit, instance-conditioned rubric decomposition, addressing the absence of a stable factorized interface for applying structured human preferences. RPO extends this structure into policy optimization, while ARR and ARR-RPO improve evaluation and generation results without judge fine-tuning or large-scale reward annotation.

  • Motivation: Human preferences span aesthetic value, semantic fidelity, and contextual appropriateness, but prevailing RLHF collapses these multidimensional judgments into scalar or latent pairwise signals.This collapse provides limited constraints on how improvement is achieved and contributes to reward-modeling vulnerabilities described in the introduction.
  • Motivation: Recent Rubrics as Reward methods recover explicit criteria but remain limited by fixed or supervised rubric construction, restricting scalability, prompt specificity, and data efficiency.The introduction notes that these limitations become more pronounced for multimodal generation.
  • ARR: ARR externalizes implicit human preferences into instance-conditioned, interpretable rubrics through training-free rubric extraction and a generate-verify-refine pipeline grounded in observable evidence.The framework is presented as a compact set of verifiable criteria for scalable multimodal evaluation requiring only a few annotated samples.
  • RPO: RPO uses ARR-generated criteria to produce binary preference decisions directly in the policy-optimization objective, preserving factorized preference structure instead of collapsing it into scalar rewards.This distinguishes RPO from prior rubric methods that use criteria only as auxiliary filters.
  • Results: 1.7 to 6.3 points: ARR improves preference accuracy over trained reward models and direct VLM judges while reducing positional bias and retaining zero-shot and few-shot generalization.ARR-RPO further improves text-to-image and image-editing benchmarks, including GenEval: 0.66 to 0.80 and DPG-Bench: 83.84 to 85.76, without judge fine-tuning or large-scale reward annotation.

2 Related Work

Prior multimodal reward models compress rich human preferences into scalar signals, enabling coarse ranking but obscuring preference structure and inviting reward hacking. Rubric-based methods provide more interpretable and stable criterion-level evaluation, though existing approaches remain limited in scope.

  • Multimodal Reward Modeling: PickScore, ImageReward, and HPS compress rich human preferences into scalar signals for multimodal alignment.These reward models support alignment across text-to-image generation, editing, and video synthesis.
  • Multimodal Reward Modeling: Scalar compression enables coarse ranking but obscures preference structure and is prone to reward hacking and overfitting.
  • Rubric as Reward: Rubric-based formulations decompose judgments into interpretable criteria, with language-task methods yielding more stable and calibrated signals than holistic scoring.Analytic rubric frameworks and LLM-Rubric demonstrate criterion-level assessment benefits.
  • Rubric as Reward: AutoRubric distills generalizable criteria from preference data but remains confined to text-only evaluation.

3 Methodology

ARR decomposes preference alignment into rubric-based evaluation and Rubric Policy Optimization, converting implicit human preferences into explicit, verifiable criteria and binary rewards. The method generates, verifies, structures, and applies rubrics online to guide policy optimization while reducing reliance on opaque scalar reward models.

  • Framework Overview: ARR decomposes preference alignment into explicit rubric-based evaluation and RPO training.The framework addresses preference-model parameterization by separating evaluation from training.
  • Explicit Preference Modeling: A rubric-conditioned VLM judge processes paired candidates and produces a binary preference decision approximating human preferences.The natural-language rubric is designed to help the judge discern subtle differences in response quality.
  • Verifiable Rubric Generation: ARR generates instance-specific rubrics, independently verifies their consistency with preference labels, and refines or discards failed rubrics.Verification provides a safeguard against self-reinforcing errors and idiosyncratic model biases.
  • Hierarchical Rubric Structuring: The verified rubric collection is consolidated into a compact, hierarchically organized evaluation protocol for consistent conditioning across prompts.An LLM groups related criteria by semantic granularity and preference dimension, producing Rstructured for reuse as a system prompt.
  • Rubric Policy Optimization: RPO uses rubric-conditioned binary judgments as online rewards, assigning +λ to preferred outputs and −γ to dispreferred outputs.The winning and losing trajectory advantages are distributed uniformly across generation or denoising timesteps, while frozen rubric judges help mitigate reward hacking.

4 Experiments

ARR is evaluated as both a preference evaluator and structured reward across multimodal understanding, text-to-image generation, and image-editing benchmarks. Rubric conditioning improves evaluator reliability, while ARR-RPO transfers these gains into generative quality and generalizes across model families.

  • Preference Evaluation: ARR is tested on HPDV3, MM-RewardBench2, and EditReward-Bench using pairwise preference accuracy against trained reward models and VLM judges.Baselines include HPSv3, PickScore, ImageReward, UnifiedReward, UnifiedReward-Thinking, EditReward, Qwen3-VL, GPT-5, and Gemini 3.1 Pro.
  • Preference Evaluation: ARR conditioning improves evaluator accuracy by 1.7–6.3 points, while specialist reward models show narrow generalization and direct VLM judges struggle on challenging splits.HPSv3 drops from 76.9% on HPDv3 to 60.2% on MM-RewardBench2 T2I, and Gemini 3.1 Pro reaches only 61.2% on EditReward-Bench.
  • Generative Policy Optimization: ARR-RPO outperforms specialist baselines across text-to-image and editing benchmarks, improving GenEval from 0.66 to 0.80 and GEdit-Bench from 7.54 to 7.85.Other reported gains include DPG-Bench 83.84→85.76, TIIF 71.09→76.85, UniGenBench++ Short 60.97→65.89, and ImgEdit 4.35→4.43.
  • Ablations and Generalization: Transferred rubrics recover more than half of the performance gap in weaker generators, indicating effectiveness depends on rubric quality and structure rather than tight evaluator–generator coupling.ARR-RPO uses generated rubrics as binary rewards with FLUX.1.dev and Qwen-Image-Edit-2509, and examines Qwen3-VL-8B, GPT-5, and Gemini 3.1 Pro judges.
  • Ablations and Generalization: Increasing rubric dimensionality consistently improves accuracy, supporting finer-grained factorization of preference structure as a source of ARR’s gains.The ablation attributes gains to rubric factorization and content quality rather than model capacity or evaluator–generator coupling.

5 Conclusion · Appendix · A Experimental Setup Details

ARR and RPO unify multimodal preference evaluation with generative alignment by converting latent human preferences into explicit, instance-conditioned rubrics and structured reward signals. The paper also documents its experimental resources, including an 8-GPU NVIDIA H100 cluster.

  • 5 Conclusion: ARR and RPO form a unified framework connecting multimodal preference evaluation with generative alignment.The framework bridges evaluation and alignment within one pipeline.
  • 5 Conclusion: ARR prompts VLMs to externalize latent human preferences as explicit, interpretable, instance-conditioned rubrics.The rubrics are generated for each instance rather than relying only on implicit scalar signals.
  • 5 Conclusion: These rubrics provide structured, factorized reward signals for Rubric Policy Optimization.The supplied conclusion identifies rubrics as the interface connecting ARR’s evaluation structure to RPO.
  • 5 Conclusion: Prevailing implicit, entangled scalar signals obscure underlying criteria and introduce systematic biases.ARR is presented as a response to these limitations of existing approaches.
  • Appendix: The appendix details the datasets, evaluation protocols, model configurations, training hyperparameters, and computational resources used throughout the paper.The supplied experimental-setup passage characterizes this material as comprehensive.
  • A Experimental Setup Details: All experiments were conducted on a cluster of 8 NVIDIA H100 (80GB SXM5) GPUs.This specifies the computational hardware used for the experiments.

A.1 Datasets … A.4 Generative Training: RPO Hyperparameters

The experiments span preference-evaluation and generative-quality benchmarks, with explicit protocols for measuring preference accuracy and positional robustness. ARR uses frozen VLM-generated rubrics, while RPO trains diffusion policies using rubric-conditioned binary advantages.

  • A.1 Datasets: The evaluation covers preference fidelity and generative quality in text-to-image synthesis and instruction-guided image editing.These are the two benchmark families used in the study.
  • A.1 Datasets: HPDv3 contains 1.17 million text-to-image pairwise comparisons, and evaluation uses its official test split with pairwise preference accuracy.Each pair contains two images from the same prompt, with one human-preferred image.
  • A.1 Datasets: GenEval, DPG-Bench, and TIIF measure compositional object accuracy, dense-prompt alignment, and instruction fidelity, respectively.GenEval uses object detection against structured prompt decompositions, while DPG-Bench reports an overall alignment score averaged across test prompts.
  • RL Training Datasets: ShareGPT-4o-Image provides around 92K GPT-4o-synthesized text-to-image and text-guided editing samples for constructing training and evaluation prompts.The corpus is described as large-scale and high-quality.
  • A.2 Evaluation Protocols: Preference accuracy is the proportion of test pairs assigned higher reward to the human-preferred image, with forward–reverse gaps quantifying position bias.Each pair is evaluated in both original and permuted order.
  • A.3 Model Configurations: Unless otherwise specified, ARR uses a frozen VLM to synthesize five prompt-conditioned rubrics that decompose prompts into independently verifiable binary criteria.Example dimensions include object presence, attribute accuracy, spatial layout, aesthetic quality, and instruction adherence.
  • A.3 Model Configurations: ARR with guide adds held-out human-curated preference exemplars as in-context semantic anchors without fine-tuning the VLM.The exemplars are prompt, preferred image, dispreferred image, and preference-rationale tuples embedded verbatim.
  • A.4 Generative Training: RPO Hyperparameters: RPO trains denoising diffusion policies for text-to-image and image editing by comparing two online candidate outputs with a frozen ARR judge and uniformly applying A ∈{+λ, −γ} across timesteps.Training prompts are sampled uniformly from ShareGPT4o-Image without data augmentation.

B Auto-Rubric as Reward (ARR) Details · C Ablations on Position Bias in ARR · C.1 Setup

ARR converts preference judgments into explicit, prompt-conditioned rubrics through generation, verification, refinement, and hierarchical structuring, while operating as a training-free VLM judge. Its setup and design target interpretable, data-efficient evaluation and expose positional instability through forward-versus-reverse testing.

  • B.1 Rubric Generation Pipeline: ARR generates, verifies, and structures prompt-conditioned rubrics through three frozen multimodal language-model calls, leaving the judge VLM unmodified.The stages are generation, verification, and structuring.
  • B.1.1 Per-Instance Rubric Generation: For each preference pair, the generator decomposes the rationale into independent, verifiable binary criteria that avoid holistic or comparative language.Example axes include semantic fidelity, attribute accuracy, and spatial coherence.
  • B.1.2 Verification and Refinement: A verifier tests whether each rubric recovers the original preference, while failed candidates undergo critique-guided refinement for up to Tmax = 5 iterations before possible discard.87% of initial rubrics pass without refinement, and fewer than 4% are ultimately discarded.
  • B.1.3 Hierarchical Structuring: Verified rubrics are aggregated into a hierarchical prompt covering overall alignment, compositional structure, and fine-grained fidelity, then used to condition evaluation and RPO training.Each dimension groups related rubrics and includes an operationalization clause.
  • B.2 Comparative Characterization of Reward Modeling Paradigms: ARR enables zero-shot rubric generation from frozen VLMs, supports immediate deployment without task-specific supervision, and avoids judge-model parameter updates.This training-free interface removes the computational and data overhead associated with training pointwise or pairwise reward models.
  • B.2 Comparative Characterization of Reward Modeling Paradigms: ARR jointly considers rubric dimensions in a single pairwise decision and constructs high-quality criteria from as few as 100 preference pairs.Joint consideration preserves intercriterion dependencies, while the reported supervision level demonstrates data-efficient rubric induction.
  • C.1 Setup: Position bias is assessed on HPDv3 by comparing forward and reverse presentation accuracy, with ∆= Accfwd −Accrev measuring positional instability.An unbiased evaluator would achieve identical accuracy under both presentation orders.

C.2 Results

The results evaluate ARR’s effect on positional instability across three base VLMs and their ARR-augmented variants on HPDv3. ARR consistently reduces positional bias, with the guided variant providing the strongest stabilization.

  • Position bias ablation: Table 5 evaluates position bias across three base VLMs and their ARR-augmented variants on the HPDv3 test set.All experiments use the HPDv3 test set.
  • Position bias ablation: Forward and reverse accuracy measure preference changes when the order of two images is swapped.The comparison uses preference pairs.
  • Position bias ablation: ARR variants consistently reduce ∆, while ARR w/ guide achieves the strongest stabilization against positional instability.∆ = Fwd − Rev quantifies positional instability.

C.3 Analysis … D.2 Results

ARR substantially reduces positional bias, especially when conditioned on human preference guidance, while cross-model transfer shows that rubric structure—not shared model biases—drives gains. The supplied ablation setup frames rubric cardinality as a trade-off evaluated using position-balanced preference accuracy.

  • C.3 Analysis: Base VLMs show severe, scale-invariant positional bias, with forward–reverse accuracy gaps of 34.6, 32.6, and 30.2 for Qwen3-VL-8B, GPT-5, and Gemini 3.1 Pro.The smallest gap occurs for Gemini 3.1 Pro but remains operationally severe.
  • C.3 Analysis: Zero-shot ARR reduces positional-bias gaps by 3.0–4.8 points across all three models, but substantial instability remains.For Qwen3-VL-8B, the gap decreases from 34.6 to 31.6.
  • C.3 Analysis: ARR with preference guidance reduces the gaps to 10.3, 9.3, and 8.9 for Qwen3-VL-8B, GPT-5, and Gemini 3.1 Pro, respectively.These correspond to reductions of 24.3, 23.3, and 21.3 points relative to the unaugmented baselines.
  • C.3 Analysis: Qwen3-VL-8B reverse accuracy rises from 49.9% to 79.8% with ARR and preference guidance, indicating stronger order-agnostic quality discrimination.The result supports the interpretation that guidance improves criterion grounding rather than merely feature perception.
  • C.3 Analysis: Residual positional instability remains non-trivial even for Gemini 3.1 Pro with ARR and guidance, where ∆= 8.9 versus ∆= 0 for a perfectly unbiased evaluator.Human preference guidance amplifies ARR’s effect but does not eliminate the need for more stable criteria.
  • C.4 Cross-Model Rubric Transfer: With Gemini 3.1 Pro fixed as judge, Qwen3-VL-8B-generated rubrics improve HPDv3 accuracy from 75.9% to 77.5%, while same-family rubrics reach 79.2%.The cross-model gain closes more than half of the gap to same-family rubrics.
  • D.1 Setup: Rubric cardinality is varied over K ∈{1, 5, 10, 20} because too few dimensions may underspecify quality, whereas too many may add redundant, conflicting, or noisy criteria.The study applies the same hierarchical evaluation procedure across cardinalities.
  • D.2 Results: Table 7 reports preference accuracy averaged across forward and reverse evaluation conditions, preventing cardinality gains from being confounded by positional bias.The supplied passage describes the evaluation aggregation but does not include the table’s accuracy values.

D.3 Analysis · E Rubric Policy Optimization Details

The analysis shows that increasing rubric cardinality improves preference accuracy, while K = 5 offers a practical accuracy–efficiency trade-off. RPO uses ARR-generated rubric-conditioned binary rewards in an online policy-gradient procedure, with KL regularization and reduced reward variance supporting stable training.

  • D.3 Analysis: Accuracy rises monotonically from 69.8% at K = 1 to 74.4% at K = 20, a net gain of 4.6 percentage points.Additional rubric dimensions provide complementary quality information rather than merely redundant coverage.
  • D.3 Analysis: The gain is 0.4 percentage points from K = 1 to K = 5, followed by +1.9 points from K = 5 to K = 10 and +2.3 points from K = 10 to K = 20.Finer-grained decomposition becomes increasingly consequential for difficult preference pairs.
  • D.3 Analysis: K = 5 achieves 70.2% accuracy with modest computational overhead and is selected as the default for main experiments.Each additional rubric requires separate generation, verification, and evaluation calls, producing a linear inference-cost increase.
  • D.3 Analysis: At high cardinality, noisy or redundant criteria become more likely, but averaging their low-magnitude gradient signals avoids harming overall RPO convergence.The marginal rubric at K = 20 is less discriminative than the most salient rubric at K = 1.
  • E.1 Algorithm Overview: RPO is a fully online policy-gradient algorithm that samples prompts, generates two outputs, and evaluates them with a frozen ARR judge conditioned on dynamically synthesized rubrics.ARR-generated rubrics serve as binary reward signals for aligning the generative policy with multidimensional human preferences.
  • E.2 KL Regularization and Training Stability: The KL penalty preserves pretrained generative priors and limits entropy reduction to mitigate mode collapse, with β = 0.01 for T2I and β = 0.02 for image editing.The higher editing coefficient reflects the narrower action space described in the passage.
  • E.2 KL Regularization and Training Stability: RPO exhibits substantially lower reward-trajectory variance than reward-model-based RL baselines because its frozen judge avoids reward drift and rubric-conditioned binary signals provide consistent gradient directions.Scalar reward models instead collapse multidimensional quality into a single value vulnerable to distributional shift.

F Limitation · G Image Generation and Editing Examples

The work studies frozen multimodal models to isolate auto-rubric effects, showing that self-generated criteria can support interpretable, bias-resistant preference alignment without additional annotation. It also acknowledges limits in rubric precision and illustrates applications through text-to-image generation and image editing examples.

  • F Limitation: The study deliberately focuses on frozen multimodal foundation models to isolate the effect of externalizing latent preference knowledge through auto-generated rubrics.The rubrics convert implicit, entangled preferences into explicit, independently verifiable criteria.
  • F Limitation: Pairwise comparison is adopted because its comparative structure is more resistant to reward hacking than pointwise scoring or differentiable reward models.Prompt-specific rubrics further ground judgments in inspectable, independently verifiable criteria.
  • F Limitation: ARR accommodates human supervision for refining rubric quality and specificity, but the present work emphasizes alignment without additional annotation.The experiments use self-generated criteria to guide evaluation and optimization.
  • F Limitation: Auto-rubric processing enables multimodal foundation models to self-improve their comprehension and reasoning over human preferences using only self-generated criteria.The rubric mechanism provides a scalable structure that transforms latent knowledge into actionable, verifiable constraints.
  • F Limitation: Fully automated rubric generation may still lack the precision and domain-specific nuance that curated human guidance could provide.The paper therefore treats deeper integration of human supervision as an acknowledged limitation.
  • G Image Generation and Editing Examples: Figure 5 presents examples of text-to-image generation.The figure is included as a visual example of the generation setting.
  • G Image Generation and Editing Examples: Figure 6 presents examples of image editing.The figure is included as a visual example of the editing setting.

H Full Results … I Prompts and Rubrics

The full-results sections report generative, post-training, and human-preference evaluations, while the prompts-and-rubrics section illustrates ARR’s prompt-conditioned criteria and evaluation prompts for text-to-image generation and image editing.

  • H Full Results: Table 8 reports generative performance across text-to-image and image-editing benchmarks, marking ARR-RPO rows and absolute gains over the baseline.Blue-shaded rows denote ARR-RPO, and green arrows indicate absolute baseline gains.
  • H.1 Image Generation and Editing: Table 9 reports BAGEL post-training performance using ARR-RPO across text-to-image benchmarks, highlighting ARR-RPO variants and gains over the BAGEL baseline.The best result in each comparison is bold, while green arrows indicate absolute gains over the BAGEL baseline.
  • H.2 Human Preference: Table 10 evaluates evaluator accuracy across four preference benchmarks by measuring agreement with human preference labels.The best result in each column is bold, and ARR rows plus absolute baseline gains are specially marked.
  • I Prompts and Rubrics: For text-to-image evaluation, ARR uses Gemini 3.1 Pro to generate prompt-conditioned rubrics spanning architectural and structural fidelity.The rubric assesses geometric accuracy, consistency of architectural details, material rendering, and artificial-surface artifacts.
  • I Prompts and Rubrics: The text-to-image evaluation system prompt instructs a VLM judge to perform pairwise comparisons with specified task, output-format, and anti-position-bias requirements.These instructions define how the judge compares generated images while addressing positional bias.
  • I Prompts and Rubrics: For image editing, ARR uses Gemini 3.1 Pro to generate rubrics centered on preserving built-structure fidelity after edits.Criteria include geometric accuracy, logical architectural details, material integrity, lighting consistency, and artifact elimination.
  • I Prompts and Rubrics: The image-editing evaluation prompt directs pairwise VLM comparison and uses Image BASE as the ground-truth reference for fidelity assessment.The prompt template specifies the comparison procedure for edited images.
Loading 2605.08354v1…