PaperlayerView exact revision on arXiv

Source-linked AI summary

DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verification

Rui Liu, Dian Yu, Zhenwen Liang, Yucheng Shi, Tong Zheng, Runpeng Dai, Haitao Mi, Pratap Tokekar, Leoweiliang

arXiv:2605.09269v1cs.CLcs.CV

TL;DR

Multimodal reward models can rely on textual priors instead of verifying fine-grained visual evidence. DeltaRubric uses planning and checklist-based verification, improving VL-RewardBench overall accuracy by +22.6 points for Qwen3-VL 4B base models.

  • Problem

    Single-step multimodal evaluators may rely on textual priors instead of checking instance-specific visual discrepancies, motivating structured visual verification.

  • Method

    DeltaRubric uses a Disagreement Planner to generate an instance-specific checklist and a Checklist Verifier to execute it, jointly optimized with multi-role reinforcement learning.

  • Results

    Across multimodal benchmarks, DeltaRubric improves base-model evaluation accuracy; on VL-RewardBench, Qwen3-VL 4B gains +22.6 points in overall accuracy.

  • Takeaways & Limitations

    Structured, verifiable evaluation steps improve multimodal reward modeling and outperform no-rubric and static-rubric baselines.

Abstract

from arXiv · show

Aligning Multimodal Large Language Models (MLLMs) requires reliable reward models, yet existing single-step evaluators can suffer from lazy judging, exploiting language priors over fine-grained visual verification. While rubric-based evaluation mitigates these biases in text-only settings, extending it to multimodal tasks is bottlenecked by the complexity of visual reasoning. The critical differences between responses often depend on instance-specific visual details. Robust evaluation requires dynamically synthesizing rubrics that isolate spatial and factual discrepancies. To address this, we introduce $\textbf{DeltaRubric}$, an approach that reformulates multimodal preference evaluation as a plan-and-execute process within a single MLLM. DeltaRubric operates in two steps: acting first as a $\textit{Disagreement Planner}$, the model generates a neutral, instance-specific verification checklist. Transitioning into a $\textit{Checklist Verifier}$, it executes these self-generated checks against the image and question to produce the final grounded judgment. We formulate DeltaRubric as a multi-role reinforcement learning problem, jointly optimizing planning and verification capabilities. Validated on Qwen3-VL 4B and 8B Instruct models, DeltaRubric achieves solid empirical gains. For instance, On VL-RewardBench, it improves base model overall accuracy by $\textbf{+22.6}$ (4B) and $\textbf{+18.8}$ (8B) points, largely outperforming standard no-rubric baselines. The results demonstrate that decomposing evaluation into structured, verifiable steps leads to more reliable and generalizable multimodal reward modeling.

1 Introduction

DeltaRubric addresses unreliable multimodal reward modeling by decomposing preference evaluation into instance-specific planning and visual verification within one MLLM. A multi-role reinforcement learning formulation jointly optimizes these capabilities and yields substantial gains on multimodal reward benchmarks.

  • Motivation: Multimodal single-step evaluators can bypass fine-grained visual verification by exploiting textual priors or length biases.This creates a need to assess textual claims against visual evidence rather than directly producing holistic judgments.
  • Motivation: Instance-specific visual differences, including object counts, spatial relationships, and localized hallucinations, make multimodal rubric evaluation difficult.The critical response differences often depend on highly specific visual details, creating a bottleneck in visual reasoning.
  • DeltaRubric: DeltaRubric reformulates evaluation as plan-and-execute: a Disagreement Planner generates a neutral verification checklist, then a Checklist Verifier executes it against the image and question.Both roles are performed sequentially by a single shared MLLM.
  • Training: DeltaRubric formulates planning and verification as a multi-role reinforcement learning problem with distinct yet coordinated objectives.The Planner targets checklists that expose and correct Verifier blind spots, enabling structured, verification-driven reasoning.
  • Results: +22.6 (4B) and +18.8 (8B) points in overall accuracy on VL-RewardBench are achieved over the base models.DeltaRubric also surpasses the no-rubric baselines by +4.3 (4B) and +8.1 (8B), respectively.

2 Related Work

Prior work extends LLM-as-a-judge and scalar-reward approaches to multimodal reward modeling, while text-only research increasingly uses rubric-based evaluation. However, multimodal rubric rewards remain underexplored because visual judgments require instance-specific verification, motivating DeltaRubric’s joint optimization of rubric generation and preference evaluation.

  • Multimodal Reward Modeling: Multimodal reward modeling has adapted LLM-as-a-judge methods and pursued both direct scalar rewards and generative models with CoT reasoning.These efforts target evaluation of textual claims against visual inputs and improved reliability.
  • Multimodal Reward Modeling: Monolithic multimodal evaluators can bypass rigorous image verification by exploiting language priors, formatting, or length biases.This extends lazy-judging concerns from text-based LLM evaluation to fine-grained visual grounding.
  • Rubrics as Rewards: Text-only evaluation increasingly uses rubrics and checklists to decompose complex judgments into explicit criteria, reducing cognitive load and improving reward-model alignment.Recent approaches also scale these ideas through alternating reinforcement learning and self-evolution.
  • Rubrics as Rewards: Multimodal rubric-based rewards remain underexplored because visual evaluation requires verifying instance-level realities such as localized hallucinations, object counts, and spatial relationships.These requirements differ from text evaluation because the relevant discrepancies are highly specific to each instance.
  • Rubrics as Rewards: Unlike approaches using separate rubric-generation and preference-evaluation models, DeltaRubric jointly optimizes both capabilities through multi-role reinforcement learning.Decoupled advantage estimation encourages active search for critical visual discrepancies without cross-task variance corrupting the learning signal.

3 Approach

DeltaRubric reframes multimodal reward modeling as a self-guided, two-step process in which a shared MLLM plans neutral, disagreement-focused checks and then verifies them against the image before judging candidates. Planner and Verifier capabilities are jointly optimized through multi-role reinforcement learning with task-specific training signals.

  • Two-step evaluation: DeltaRubric uses a shared MLLM first to generate a disagreement-focused verification checklist and then to execute it for the final candidate judgment.This replaces direct scalar or binary evaluation with sequential planning and verification within one model.
  • Disagreement Planner: The Planner identifies concrete visual and factual disagreements, including attributes, object counts, spatial relations, and hallucinated claims, while remaining neutral.Its checklist is designed to be short, verifiable, and evidence-seeking rather than favoring either response.
  • Checklist Verifier: The Verifier evaluates each checklist item against the image, ignores hallucinated checks, and aggregates the resulting evidence into reasoning and a final verdict.It conditions on the original input and generated checklist before producing its judgment.
  • Multi-role reinforcement learning: Multi-role reinforcement learning jointly trains planning and verification while computing their advantages independently to preserve task-specific learning signals.Planner rewards reflect whether checklists improve over no-rubric decisions, while Verifier rewards combine final accuracy with bonuses for checklist-guided improvement.

4 Experiments

Experiments train DeltaRubric with joint Planner–Verifier reinforcement learning and evaluate it against zero-shot, no-rubric, and external-model baselines on multimodal reward benchmarks. Results show stronger verifier accuracy, increasingly decision-useful checklists, and gains on visually dependent tasks, while ablations support instance-specific planning and the proposed reward formulation.

  • Implementation Details: Training uses GRPO with Qwen3-VL-4B and 8B Instruct models, sampling N = 5 Planner checklists and M = 5 Verifier trajectories per prompt.The verifier guidance bonus coefficient is λ = 0.4.
  • Training Dynamics: DeltaRubric achieves higher Verifier training and validation accuracy than the no-rubric baseline, while Planner probe accuracy steadily increases during training.Planner probe accuracy measures the fraction of sampled checklists that guide a lightweight verdict probe to the correct winner, serving as a proxy for checklist quality.
  • VL-RewardBench: DeltaRubric achieves the best performance across all reported VL-RewardBench evaluation aspects under greedy decoding.The evaluation reports subcategory accuracy, overall accuracy, and macro-average; the method attributes its gains to targeted disagreement checklists that support faithful visual verification.
  • Multimodal RewardBench: 73.2 overall accuracy is achieved by DeltaRubric on Multimodal RewardBench for the 8B model, improving over the 68.7 DPO baseline.For the 8B model, VQA accuracy improves from a base of 71.4 to 84.9, while Safety accuracy reaches 93.3.
  • Planner Reward Ablation: 80.1 Overall accuracy with the relative Planner reward exceeds 77.6 under absolute-verdict reward, while Reasoning decreases from 72.6 to 69.1.The ablation reports degradation across all metrics when the Planner is rewarded strictly by absolute final-verdict accuracy.
  • Generalization and Ablations: DeltaRubric improves text-only overall accuracy to 84.6 and Reasoning accuracy from 79.0 to 85.7, while text-only planning remains slightly worse on average than full DeltaRubric.These ablations indicate benefits from structured checklist generation even without visual input and from using the reference image during planning.

5 Conclusions

DeltaRubric reframes multimodal evaluation as an active, two-step visual investigation that separates disagreement planning from checklist verification. Its multi-role reinforcement learning design aims to reduce lazy judging by grounding judgments in visual evidence rather than textual priors.

  • 5 Conclusions: DeltaRubric decomposes evaluation into a Disagreement Planner and a Checklist Verifier for structured visual investigation.The approach reframes multimodal evaluation as an active, two-step process.
  • 5 Conclusions: The two-role design isolates factual contradictions and grounds judgments in visual evidence instead of textual priors.This is intended to mitigate lazy judging and improve evaluation reliability.
  • 5 Conclusions: DeltaRubric is optimized through multi-role reinforcement learning.The supplied passage identifies this optimization as part of the method’s design.

A Appendix · A.1 Implementation Details

The appendix describes direct RL training for Qwen3-VL-4B and 8B Instruct models, using sampled planner checklists and verifier reasoning trajectories under fixed optimization and rollout settings.

  • A.1 Implementation Details: Direct RL training is conducted on Qwen3-VL-4B and 8B Instruct models.The implementation builds on the EasyR1 framework.
  • A.1 Implementation Details: The Planner samples N = 5 candidate checklists per prompt.This defines the checklist-generation sampling budget.
  • A.1 Implementation Details: The Verifier samples M = 5 reasoning trajectories per prompt.This defines the verification-generation sampling budget.
  • A.1 Implementation Details: All models are trained for 120 steps with AdamW, learning rate 1 × 10^-6, and weight decay 0.01.These are the reported optimization settings.
  • A.1 Implementation Details: Training uses a global batch size of 128 and a rollout batch size of 256.The two batch sizes are specified separately for training and rollouts.
  • A.1 Implementation Details: Rollouts are generated with temperature 1.0.The temperature is fixed in the implementation configuration.

A.2 Sensitivity Analysis

Sensitivity analysis shows that the verifier’s guidance bonus improves performance over no guidance, with λ = 0.4 providing the best balance on VL-RewardBench. Removing the bonus weakens visual grounding, while λ = 0.6 harms complex reasoning through excessive checklist adherence.

  • Guidance bonus coefficient: λ = 0.4 achieves the optimal balance, reaching 80.1 Overall Accuracy on VL-RewardBench.This setting also attains the highest macro average among the tested coefficients.
  • Guidance bonus coefficient: Adding a guidance bonus outperforms the unguided λ = 0.0 baseline.The analysis tests λ values of 0.0, 0.2, 0.4, and 0.6.
  • Guidance bonus coefficient: λ = 0.0 reduces visual grounding, whereas λ = 0.6 degrades complex reasoning by over-optimizing checklist adherence.The findings identify distinct failure modes at the lowest and highest tested coefficients.

A.3 Generalization to Alternative RL Algorithm.

DeltaRubric is agnostic to the underlying reinforcement-learning algorithm beyond GRPO. The framework integrates DAPO through decoupled advantage estimation that separately isolates Planner and Verifier reward signals for joint policy updates.

  • Generalization to Alternative RL Algorithm: DeltaRubric remains applicable when replacing GRPO with the alternative DAPO algorithm.The passage explicitly presents DAPO as an alternative algorithm used to demonstrate the framework’s algorithmic agnosticism.
  • Generalization to Alternative RL Algorithm: Decoupled advantage estimation isolates Planner and Verifier reward signals, enabling DAPO to govern joint policy updates.This separation allows DAPO to be integrated seamlessly into the joint optimization procedure.

A.4 Discussions and Future Work

DeltaRubric provides a structurally grounded framework for multimodal evaluation while motivating extensions that improve efficiency and broaden its modality coverage. Future directions include selectively generating checklists for ambiguous cases and extending verification to video.

  • DeltaRubric establishes a robust, structurally grounded framework for multimodal evaluation.
  • Dynamic routing could trigger checklist generation only for highly ambiguous cases.
  • Extending DeltaRubric to temporal modalities such as video evaluation could scale instance-specific verification.

A.5 Qualitative Examples · A.6 Prompt Templates

Qualitative examples show that DeltaRubric counters visual hallucinations, logical inconsistencies, and fine-grained attribute confusion by executing targeted verification checklists. The prompt templates specify separate no-rubric, static-rubric, DeltaRubric planner, verifier, and cheap-verdict procedures for structured evaluation.

  • A.5 Qualitative Examples: DeltaRubric mitigates visual hallucinations and flawed reasoning by enforcing rigorous visual verification through generated checklists.The qualitative comparison contrasts this process with the standard no-rubric baseline.
  • A.5 Qualitative Examples: DeltaRubric isolates conflicting shoe colors and correctly rejects Response B’s hallucinated “white shoes” in favor of Response A.The checklist targets the fine-grained visual discrepancy that the no-rubric baseline fails to verify.
  • A.6 Prompt Templates: The prompt appendix provides complete evaluation prompts for no-rubric, static-rubric, DeltaRubric, planner, and cheap-verdict configurations.It also includes the static rubrics used by the baseline evaluations.
  • A.5 Qualitative Examples: DeltaRubric verifies visual evidence about granular food in an open palm rather than a tree branch, correctly selecting Response B despite the baseline’s inconsistency.The checklist systematically links the visual evidence to the final judgment.
  • A.6 Prompt Templates: DeltaRubric verifies the box’s “green exterior,” preventing confusion with white tissues and correctly selecting Response A.The targeted checklist addresses fine-grained visual attribute binding.
  • A.6 Prompt Templates: The static-rubric prompt evaluates five criteria: question relevance, factual consistency, important visual information, reasoning, and answer clarity and completeness.It instructs the model to ignore criteria that are irrelevant, vague, or contradicted by the image or question.
  • A.6 Prompt Templates: The DeltaRubric verifier executes each checklist item, states supporting evidence, and records which response it favors while treating the checklist as executable checks rather than evidence.Irrelevant, vague, or contradicted checklist items are ignored.
  • A.6 Prompt Templates: The planner prompt produces 2-4 neutral, evidence-seeking checks focused on decisive disagreements, with each check covering one concrete fact, relation, or constraint.The planner must avoid naming Response A or Response B, stating the winner, or adding preamble and explanation.
Loading 2605.09269v1…