PaperlayerView exact revision on arXiv

Source-linked AI summary

Rationale Matters: Learning Transferable Rubrics via Proxy-Guided Critique for VLM Reward Models

Weijie Qiu, Dai Guan, Junxin Wang, Zhihang Li, Yongbo Gai, Mengyu Zhou, Erchao Zhao, Xiaoxi Jiang, Guanjun Jiang

arXiv:2603.16600v2cs.CV

TL;DR

GRM rubrics are rarely optimized directly, and external judge checks are costly and do not provide a differentiable training reward. Proxy-GRM uses an independent proxy’s ability to judge from a rubric as an RL reward. It reports state-of-the-art performance across three benchmarks with about 4× less training data than comparable methods, and rubrics transfer to unseen evaluators.

  • Problem

    GRM training optimizes final verdicts while leaving rubrics unverified, and external judge checks are costly and cannot provide a differentiable reward for the GRM’s training loop.

    Source: §1 · pp. 1–2

  • Method

    Proxy-GRM uses an independent proxy’s preference accuracy from the query, response pair, and candidate rubric as a rubric-quality reward during policy-model reinforcement learning.

    Source: §3 · p. 5, §3.2 · p. 5

  • Results

    Proxy-GRM achieves state-of-the-art results across three reward benchmarks with about 4× less training data than comparable methods; on VL-RewardBench, it reaches 75.22% overall accuracy versus 73.8% for Unified-Reward-Think.

    Source: §4.2 · p. 7, §5 · p. 11

  • Takeaways & Limitations

    The learned rubrics transfer to unseen evaluator models without additional training.

    Source: §1 · pp. 2–3

    An insufficiently capable proxy can introduce harmful training noise; the 3B proxy performed below the no-proxy baseline on MM-RLHF-Reward Bench, at 76.48% versus 80.59%.

    Source: §4.3 · p. 9

Abstract

from arXiv · show

Generative reward models (GRMs) for vision-language models (VLMs) often evaluate outputs via a three-stage pipeline: rubric generation, criterion-based scoring, and a final verdict. However, the intermediate rubric is rarely optimized directly. Prior work typically either treats rubrics as incidental or relies on expensive LLM-as-judge checks that provide no differentiable signal and limited training-time guidance. We propose Proxy-GRM, which introduces proxy-guided rubric verification into Reinforcement Learning (RL) to explicitly enhance rubric quality. Concretely, we train lightweight proxy agents (Proxy-SFT and Proxy-RL) that take a candidate rubric together with the original query and preference pair, and then predict the preference ordering using only the rubric as evidence. The proxy's prediction accuracy serves as a rubric-quality reward, incentivizing the model to produce rubrics that are internally consistent and transferable. With ~50k data samples, Proxy-GRM reaches state-of-the-art results on the VL-Reward Bench, Multimodal Reward Bench, and MM-RLHF-Reward Bench, outperforming the methods trained on four times the data. Ablations show Proxy-SFT is a stronger verifier than Proxy-RL, and implicit reward aggregation performs best. Crucially, the learned rubrics transfer to unseen evaluators, improving reward accuracy at test time without additional training. Our code is available at https://github.com/Qwen-Applications/Proxy-GRM.

1. Introduction

Proxy-GRM adds proxy-based rubric verification to GRM training, where rubrics are otherwise left unsupervised despite their importance to reliable evaluation. Its results and transfer tests indicate that proxy-guided rubrics can improve reward-model performance and work with unseen evaluators.

1. Introduction

  • Generative reward models produce natural-language critiques alongside preference verdicts, commonly decomposing evaluation into a rubric, criterion-by-criterion assessment, and final answer.This structure provides interpretability, but the reliability of later stages depends on rubric quality.Source: §1 · p. 1
  • Existing GRM training optimizes the final verdict while leaving rubrics unverified, so models may rationalize a predetermined answer and produce criteria that fail with independent evaluators or different task distributions.LLM-as-a-Judge checks are costly at training scale and provide no differentiable reward to close the training loop.Source: §1 · pp. 1–2
  • Figure 1 contrasts final-answer-only training with proxy-guided rubric optimization and shows Proxy-GRM using 1× data outperforming baselines trained with at least 4× more data.The figure compares results on VL-RewardBench, MultiModal Reward Bench, and MM-RLHF-Reward Bench.Source: Figure 1 · p. 2
  • Proxy-GRM reaches state-of-the-art results on three reward benchmarks with about 50k training samples, roughly 4× less data than comparable methods, and its rubrics transfer to unseen evaluators.Proxy-SFT outperforms Proxy-RL as a verifier.Source: §1 · pp. 2–3

2. Related Work

Prior work includes scalar and generative reward models, rubric-based evaluation, and RL approaches to reward modeling and process supervision. Proxy-GRM differs by verifying rubric transferability with an external proxy and incorporating that signal into multimodal RL.

2.1. Reward Models for Vision-Language Models

  • Generative reward models provide interpretable critiques alongside preference judgments, but prior models generally do not explicitly verify or optimize generated rubric quality.Earlier multimodal approaches include R1-Reward [Zhang et al., 2025a] and Unified-Reward [Wang et al., 2025b].Source: §2.1 · p. 3

2.2. Rubric-Based Evaluation and Reward Modeling

  • Rubric-based work spans learned rubric weights, rubric-structured RL rewards, and alternating rubric refinement, while Proxy-GRM adds external verification of rubric transferability within the RL loop.Auto-Rubric [Xie et al., 2025] extracts rubrics from annotations; Rubrics-as-Rewards [Gunjal et al., 2025] scores subcriteria; Xu et al. [2026] alternate rubric refinement and reward-model training.Source: §2.2 · p. 3

2.3. Reinforcement Learning for Language and Vision-Language Models

  • Prior RL work trains multimodal reward models and evaluates intermediate reasoning, whereas this approach adds a proxy reward for rubric quality alongside final-output supervision.The related work frames this as bridging outcome and process supervision [Lightman et al., 2023; Uesato et al., 2022].Source: §2.3 · p. 4

2.4. Process Supervision and Verification

3. Methods

Proxy-GRM trains rubrics to guide independent evaluators toward the correct preference, using proxy accuracy as a rubric-transferability signal. Its training combines proxy-agent preparation, supervised initialization, proxy-guided RL, and optional inference-time proxy checking.

3. Methods

  • Proxy-GRM combines proxy-agent training for rubric evaluation with proxy-guided reinforcement learning for the policy model.The method also specifies a separate inference procedure.Source: §3 · p. 5

3.1. Problem Formulation

  • Rubric transferability means an independent evaluator can use the rubric with the image, query, and response pair to reach the correct preference; proxy accuracy operationalizes this property.The proxy consumes the rubric and returns an evaluation and verdict without generating a rubric itself.Source: §3.1 · p. 5, §3.2 · p. 5

3.2. Proxy Agent Training

  • The proxy is trained only to follow supplied rubrics: Proxy-SFT uses correctly distilled examples, and Proxy-RL further trains from that checkpoint using verdict accuracy as its sole reward.This specialization is intended to keep the proxy an independent judge of rubric quality.Source: §3.2 · p. 5

3.3. Policy Model Training

  • After supervised fine-tuning on correctly distilled samples, Proxy-GRM uses GRPO with accuracy, proxy-transferability, and format rewards; the proxy remains frozen during policy training.Accuracy rewards correct verdicts, proxy reward tests whether the rubric enables a correct verdict, and format reward checks prescribed XML structure.Source: §3.3 · p. 6

3.4. Inference with Proxy Verification

  • At inference, standard mode uses the policy model’s verdict directly, while proxy-verified mode accepts agreement with the proxy as high confidence and logs disagreements without replacing the policy verdict.The proxy is not required in standard mode.Source: §3.4 · p. 6

4. Experiments

Proxy-GRM combines benchmark comparisons with analyses of proxy selection, reward feedback, and rubric transferability. Its strongest reported results use Proxy-SFT, and generated rubrics improve accuracy for external evaluators.

4.1. Implementation Details

  • Experiments compare Proxy-GRM with closed-source models, open-source reward models, and ablations across three multimodal reward benchmarks.The benchmarks cover general multimodal understanding, diverse reward tasks, and MM-RLHF evaluation tasks; the policy and proxy agents use Qwen2.5-VL-7B.Source: §4 · p. 6, §4.1 · pp. 6–7

4.2. Main Results

  • 75.22% overall accuracy on VL-RewardBench, 85.62% overall score on Multimodal Reward Bench, and 82.94% Acc on MM-RLHF-Reward Bench are Proxy-GRM-RL with Proxy-SFT’s reported best results.The comparisons include a 1.42-point gain over Unified-Reward-Think on VL-RewardBench and a 3.42-point gain over R1-Reward on Multimodal Reward Bench.Source: Table 1 · p. 7, §4.2 · pp. 7–8

4.3. Analysis Studies

  • Figure 3 illustrates that Proxy-SFT produces a more discriminative rubric with balanced weights and a correct prediction, while the 3B proxy fails to correct repetition.The 32B model adds a more informative criterion, but its understanding of critical rubric aspects is weaker than Proxy-SFT’s.Source: Figure 3 · p. 8, §4.3 · p. 10

4.2. Main Results

  • Proxy-GRM uses 50k samples, roughly 4× less than R1-Reward and Unified-Reward-RL, while consistently outperforming those baselines.The authors suggest proxy-guided rubric verification provides a more efficient learning signal than simply scaling training data.Source: §4.2 · p. 8

4.3. Analysis Studies

  • Proxy-SFT outperforms Proxy-RL on all three benchmarks; general-model scale helps, but Proxy-SFT’s 7B model beats Qwen2.5-VL-32B-Instruct on VL-RewardBench.The 3B proxy falls below the no-proxy MM-RLHF-Reward Bench baseline, while the dedicated Proxy-SFT agent reaches 75.22% versus 74.02% for the 32B model.Source: §4.3 · pp. 9–10
  • Implicit additive feedback outperforms explicit reward modifications; the most aggressive penalty scheme causes training instability.Moderate schemes underperform the additive baseline by 1–3 points.Source: §4.3 · p. 10

4.4. Rubric Transferability Analysis

  • Across three benchmarks and three evaluator models, Proxy-SFT rubrics provide the best transfer accuracy; Unified-Reward-SFT gains 3.76 points on VL-RewardBench using Proxy-GRM-RL rubrics.The transfer protocol tests rubrics generated by different sources with external evaluators, measuring preference accuracy.Source: §4.4 · p. 10, Table 4 · p. 11

5. Conclusion

Proxy-GRM uses proxy agents to verify rubric quality during RL, improving reward-model performance and rubric transferability. The supplementary material also includes the prompts and data preparation used in the framework.

5. Conclusion

  • Proxy-GRM improves reward-model performance across three benchmarks with 4× less training data, Proxy-SFT outperforms Proxy-RL, implicit reward aggregation works better, and rubrics transfer to unseen evaluators.The conclusion identifies external verifiability of intermediate reasoning artifacts as a promising direction for improving reward-model reliability and interpretability.Source: §5 · p. 11

A.1. Instruction Prompt

  • Figure 5 presents the system prompt for distillation and RL.Source: Figure 5 · p. 16

A.2. Rubric Evaluation Prompt

B.1. Data Sources and Curation

  • The training data comes from four multimodal preference datasets and is filtered for quality, difficulty, and semantic similarity.The curated set contains approximately 60k samples after filtering.Source: §B.1 · p. 17

B.2. Data Distillation

  • Qwen3-VL-235B-A22B generates structured rubric, assessment, and verdict outputs for each curated sample.The distillation prompt directs the teacher to produce fine-grained criteria, apply them systematically, and give a final verdict.Source: §B.2 · p. 17

B.3. Data Allocation

The training allocation combines correct teacher outputs with hard samples for RL, producing a 50k-sample pipeline that is roughly four times smaller than comparable methods.

B.3. Data Allocation

  • Approximately 25k teacher-correct outputs are allocated separately, while the remaining 35k samples, including incorrect teacher outputs, serve as hard negatives for RL training.Source: §B.3 · p. 17, Table 5 · p. 17
  • 50k samples power the full Proxy-GRM-RL pipeline, roughly 4× fewer than the >200k samples used by R1-Reward and Unified-Reward-Think.Source: §B.3 · p. 17

C. Training Details

The training details specify separate SFT and RL implementations and align proxy-agent training with the corresponding policy-model stages. The appendix also provides benchmark-level result breakdowns supplementing the main table.

C.1. Proxy-GRM-SFT

  • The policy model’s cold-start SFT stage uses ms-swift.Source: §C.1 · p. 17

C.2. Proxy-GRM-RL

  • The policy model’s reinforcement-learning stage uses verl with GRPO.Source: §C.2 · p. 17
  • Proxy-SFT follows the policy cold-start SFT setup, while Proxy-RL follows the Proxy-GRM-RL GRPO setup.Source: §C.2 · p. 18

D. Detailed Experimental Results

D.1. Results on VL-RewardBench

Proxy-GRM-RL with Proxy-SFT achieves the highest reported overall VL-RewardBench accuracy, with a particularly large advantage on hallucination detection and smaller gains on reasoning.

D.1. Results on VL-RewardBench

  • VL-RewardBench contains 1,247 preference pairs across general multimodal understanding, hallucination detection, and visual reasoning, assessed with overall and macro accuracy.Source: §D.1 · p. 18
  • 75.22% overall accuracy is the best result, exceeding Unified-Reward-Think by 1.42 points and R1-Reward by 3.30 points.Source: §D.1 · p. 18
  • 93.08% versus 85.71% for R1-Reward on hallucination detection, while reasoning gains are less pronounced.The passage attributes the hallucination-category pattern to criteria especially sensitive to factual grounding and notes that transferability provides most value when fine-grained criteria matter.Source: §D.1 · p. 18

D.2. Results on Multimodal Reward Bench

Proxy-GRM exceeds prior methods on Multimodal Reward Bench, with Proxy-GRM-RL plus Proxy-SFT or Unified-Reward-SFT achieving 85.6% overall accuracy.

D.2. Results on Multimodal Reward Bench

  • 85.6% overall accuracy for Proxy-GRM-RL + Proxy-SFT / Unified-Reward-SFT is 3.4 points above R1-Reward (82.2%) and 13–15 points above closed-source models.The paper identifies this as its strongest configuration on the benchmark.Source: §D.2 · p. 18
  • Both Proxy-GRM-SFT and Proxy-GRM-RL exceed all prior methods by a substantial margin on this benchmark.Source: §D.2 · p. 18
  • Performance differences among proxy-agent variants are smaller here than on VL-RewardBench, which the authors relate to the structured benchmark categories being addressable by Proxy-GRM-RL.Source: §D.2 · p. 18

D.3. Results on MM-RLHF-Reward Bench

Proxy-GRM achieves its best reported Acc on MM-RLHF-Reward Bench at 82.94%; its stricter Acc+ results and a weak-proxy degradation further distinguish configurations.

D.3. Results on MM-RLHF-Reward Bench

  • 82.94% Acc is the best result, achieved across Proxy-SFT, Unified-Reward-SFT, and R1-Reward proxy configurations.Source: §D.3 · p. 18
  • On the stricter Acc+ metric, Proxy-GRM-RL with R1-Reward achieves 63.04%, matching Claude-3.7-Sonnet and substantially outperforming R1-Reward, which scores 54.35%.Source: §D.3 · p. 18
  • The Qwen2.5-VL-3B proxy yields 76.47%, below the SFT-only baseline, consistent with harmful training noise from an insufficiently capable proxy.Source: §D.3 · p. 19

E.1. Multi-Agent Proxy Configurations

Ensembling proxy agents underperforms the best individual proxy across benchmarks; the authors attribute this to conflicting, noisy reward signals.

E.1. Multi-Agent Proxy Configurations

  • Multi-agent proxy rewards average individual rewards without weights, using pairwise and triple combinations of four strong agents.The agents are Proxy-SFT, Unified-Reward-SFT, Qwen2.5-VL-7B-Instruct, and Qwen2.5-VL-32B-Instruct.Source: §E.1 · p. 19
  • 62.71% on VL-RewardBench for the best ensemble, PSFT + Q32B, is 12.51 points below PSFT alone at 75.22%.The degradation trend is consistent across all three benchmarks.Source: §E.1 · p. 21, Figure 7 · p. 21
  • Early proxy-reward optimization shows high variance and occasional oscillations, while differing evaluator capabilities and biases produce conflicting signals.The authors say averaging can wash out informative gradients and hinder coherent rubric learning.Source: Figure 7 · p. 21, §E.1 · p. 21
  • The authors conclude that a single well-calibrated proxy is preferable to heterogeneous agents whose evaluation tendencies are misaligned.Source: §E.1 · p. 21

E.2. Ablation on Proxy-SFT Training Data Size

The data-size ablation favors 5k Proxy-SFT samples, while qualitative cases show stronger proxies producing more task-aligned rubrics and better preference judgments.

E.2. Ablation on Proxy-SFT Training Data Size

  • Because proxy judgment lacks a direct evaluation metric, the ablation tests 2.5k, 5k, 7.5k, and 10k samples through downstream Proxy-GRM-RL performance.Other hyperparameters are held fixed.Source: §E.2 · p. 21
  • 5k Proxy-SFT samples perform best across benchmarks except Acc+, where 2.5k and 5k tie; performance degrades with 7.5k or 10k.The authors hypothesize that lower-quality or ambiguous additional samples introduce noise and miscalibrate proxy rewards.Source: §E.2 · pp. 21–22, Table 9 · p. 22

F. More Case Studies

  • Across object-description, action-understanding, and game-state cases, Proxy-SFT produces more balanced, non-redundant, task-specific rubrics and more reliable preference judgments.Smaller proxies often retain redundancy or miss key errors, while the stronger Qwen2.5-VL-32B proxy improves task alignment.Source: F. More Case Studies · p. 22

Related papers

Loading 2603.16600v2…