You Only Judge Once: Multi-response Reward Modeling in a Single Forward Pass

Yinuo Yang; Zixian Ma; Manasi Ganti; Jieyu Zhang; Ranjay Krishna

Source-linked AI summary

You Only Judge Once: Multi-response Reward Modeling in a Single Forward Pass

Yinuo Yang, Zixian Ma, Manasi Ganti, Jieyu Zhang, Ranjay Krishna

arXiv:2604.10966v2Apr 13, 2026cs.CVcs.AI

TL;DR

Existing multimodal reward models require repeated or autoregressive scoring and benchmarks largely test only pairwise comparisons, limiting efficient N-way preference evaluation. The paper introduces a single-pass multi-response discriminative reward model and two N-way benchmarks. It achieves state-of-the-art results across six multimodal reward benchmarks and improves GRPO training and open-ended generation quality over a single-response baseline.

Problem
Existing multimodal reward models and benchmarks are limited for efficient N-way candidate scoring, despite N-way comparisons arising in ranking and policy optimization.
Method
The model concatenates N responses, extracts per-response representations, scores them with a value head, and trains using cross-entropy over their scalar scores in one forward pass.
Results
The model achieves state-of-the-art results across six multimodal reward benchmarks and improves open-ended generation quality and GRPO training behavior over a single-response RM baseline.
Takeaways & Limitations
Single-pass multi-response scoring supports scalable preference evaluation and stronger downstream GRPO policy optimization within the demonstrated multimodal settings.
Takeaways & Limitations
Video preference evaluation remains challenging, larger N values were not evaluated, and the model cannot provide natural-language rationales.

Abstract

from arXiv · show

We present a discriminative multimodal reward model that scores all candidate responses in a single forward pass. Conventional discriminative reward models evaluate each response independently, requiring multiple forward passes, one for each potential response. Our approach concatenates multiple responses with separator tokens and applies cross-entropy over their scalar scores, enabling direct comparative reasoning and efficient $N$-way preference learning. The multi-response design also yields up to $N\times$ wall-clock speedup and FLOPs reduction over conventional single-response scoring. To enable $N$-way reward evaluation beyond existing pairwise benchmarks, we construct two new benchmarks: (1) MR$^2$Bench-Image contains human-annotated rankings over responses from 8 diverse models; (2) MR$^2$Bench-Video is a large-scale video-based reward benchmark derived from 94K crowdsourced pairwise human judgments over video question-answering spanning 19 models, denoised via preference graph ensemble. Both benchmarks provide 4-response evaluation variants sampled from the full rankings. Built on a 4B vision-language backbone with LoRA fine-tuning and a lightweight MLP value head, our model achieves state-of-the-art results on six multimodal reward benchmarks, including MR$^2$Bench-Image, MR$^2$Bench-Video, and four other existing benchmarks. Our model outperforms existing larger generative and discriminative reward models. We further demonstrate that our reward model, when used in reinforcement learning with GRPO, produces improved policy models that maintain performance across standard multimodal benchmarks while substantially improving open-ended generation quality, outperforming a single-response discriminative reward model (RM) baseline by a large margin in both training stability and open-ended generation quality.

1 Introduction

Multimodal reward models struggle with efficient N-way comparison because conventional discriminative scoring repeats forward passes and generative judging relies on autoregressive decoding. The paper introduces a single-pass multi-response model and two N-way benchmarks, achieving strong benchmark and GRPO results.

Motivation: Conventional discriminative and generative reward models do not scale gracefully to N-way ranking scenarios.These scenarios arise in best-of-N sampling and group-based policy optimization.
Method: The proposed model concatenates all N responses, produces scalar scores with a value head, and trains with cross-entropy over the scores in one forward pass.Causal attention lets each response attend to preceding responses for direct comparative reasoning.
Efficiency: Up to N× wall-clock speedup and FLOPs reduction are achieved over single-response scoring while improving accuracy.
Benchmarks: MR2Bench-Image contains 240 human-annotated rankings from 8 models, while MR2Bench-Video contains 495 questions with rankings over 19 models inferred from approximately 94K pairwise judgments.
Results: The 4B model achieves state-of-the-art results across six multimodal reward benchmarks and outperforms larger generative and comparable discriminative models.
Policy Optimization: Compared with a single-response RM, the multi-response RM provides a steadily increasing GRPO validation reward signal and larger downstream gains.

2 Related Work

Prior multimodal reward-model research established preference-based evaluation and alignment, but existing benchmarks remained centered on pairwise judgments. This paper positions explicit N-way rankings as the missing evaluation setting for joint candidate scoring.

Preference Learning: Reward models are trained on human preference data and used to guide downstream policy optimization, while alternatives such as DPO bypass explicit reward modeling.
Reward Benchmarks: Multimodal reward benchmarks cover visual perception, hallucination, reasoning, safety, VQA, and video understanding but remain centered on pairwise judgments.
Research Gap: Existing benchmarks do not directly evaluate jointly scoring multiple candidates for best-of-N selection, listwise reranking, or group-based policy optimization.
Research Gap: MR2Bench-Image and MR2Bench-Video address this gap with explicit N-way rankings.

3 Method

The method concatenates all candidate responses into one causal sequence, extracts each response’s final-token representation, and maps it to a scalar score. Cross-entropy trains the scores for listwise preference prediction while reducing the passes required for N-way comparison.

Baseline: Conventional discriminative reward models score each prompt-response pair separately and train with a Bradley-Terry pairwise loss.The chosen and rejected responses define the pairwise comparison.
Multi-Response Scoring: The proposed model concatenates the prompt and N responses with separator tokens and processes the sequence in one forward pass.A unique separator token identifies response boundaries.
Multi-Response Scoring: Causal attention allows each response to attend to preceding responses and the shared prompt, enabling implicit candidate contrast.
Response Representation: Each response is represented by the hidden state at its last token, which aggregates information from the entire response without additional pooling.
Value Head: A two-layer MLP value head maps each response representation to a scalar reward score using a SiLU activation.
Training Objective: Cross-entropy over the N scores and the ground-truth best-response index provides the training objective, reducing to Bradley-Terry loss when N=2.

4 Multi-Response Multimodal RewardBench

MR2Bench-Image and MR2Bench-Video extend multimodal reward evaluation from pairwise judgments to N-way rankings across image and video tasks. The benchmarks combine human rankings or denoised pairwise preferences and evaluate best-of-N selection on four-response variants.

Benchmark Motivation: Existing multimodal reward benchmarks lack N-way ranking evaluation, motivating MR2Bench-Image and MR2Bench-Video.
MR2Bench-Image: MR2Bench-Image contains 240 consented user-interaction prompts with images across VQA, safety-related queries, and visual reasoning.Each category contributes 80 samples.
MR2Bench-Image: Eight diverse models generate responses for MR2Bench-Image, and human annotators rank all responses to form complete orderings.Four-response variants are sampled from the full rankings.
MR2Bench-Video: MR2Bench-Video covers 497 video questions over 489 videos and diverse understanding tasks including temporal reasoning, action recognition, and visual detail comprehension.
MR2Bench-Video: Approximately 94K pairwise judgments over responses from 19 models are denoised into consistent rankings using preference graph ensemble processing.The procedure aggregates preference graphs, removes cycles, and topologically sorts the resulting DAGs.
Evaluation: Both benchmarks evaluate whether the highest-scored response matches the ground-truth rank-1 response on four-response variants.

5 Experiments

Experiments show that multi-response reward modeling improves benchmark accuracy and inference efficiency, while supporting more stable and effective GRPO policy optimization. Ablations identify cross-entropy, SiLU, and last-token pooling as strong design choices.

Benchmark results: 71.2% average accuracy across six multimodal benchmarks makes the Molmo2-4B multi-response RM the highest-scoring open-source model.The benchmarks include four image and two video reward benchmarks.
Scoring comparison: 64.8% vs. 54.0% average accuracy shows multi-response cross-entropy outperforming single-response Bradley-Terry on Molmo2-4B.On Qwen3-VL-4B, the overall gap is smaller, with cross-entropy at 65.1% versus 63.0% for Bradley-Terry.
Inference efficiency: 3.9× latency and 4.0× FLOPs reductions are achieved at N=4, with multi-response scoring using one pass instead of N single-response passes.The efficiency advantage approaches N× when visual tokens dominate because the shared visual prefix is processed once.
Inference efficiency: Multi-response latency stays nearly constant as N increases, whereas single-response cost grows linearly from N=2 to N=16.This trend is observed using MR2Bench-Video samples and also holds with the Qwen3-VL-4B backbone.
Policy optimization: GRPO with the multi-response RM improves open-ended generation while preserving performance across all 24 standard multimodal benchmarks.WildVision rises by +5.6, LLaVA-Bench by +4.6, and MMHal from 3.98 to 4.25.
Ablations: 64.8% average accuracy is obtained by both SiLU value heads and last-token pooling, while cross-entropy reaches 64.8% versus 63.8% for Plackett-Luce loss.These ablations support the adopted value-head activation, response representation, and training objective.

6 Conclusion

The paper introduces an efficient N-way multimodal reward model and benchmarks, showing strong accuracy and improved downstream GRPO behavior while identifying unresolved evaluation and interpretability limits.

Efficiency and accuracy: Up to N× wall-clock speedup and FLOPs reduction are achieved by scoring all N responses in one forward pass.The model uses a 4B backbone and achieves state-of-the-art accuracy across six benchmarks.
Downstream policy optimization: The model improves open-ended generation quality in GRPO while preserving standard multimodal benchmark performance.Its validation reward increases steadily during training, unlike the single-response baseline.
Benchmarks: MR2Bench-Image and MR2Bench-Video provide N-way ranking benchmarks intended to support scalable multimodal preference evaluation.These benchmarks address the infrastructure gap for N-way comparison.
Limitations: 50.7% best-of-4 accuracy on MR2Bench-Video shows that video preference evaluation remains challenging.Experiments evaluate only up to N=4, larger-N scaling is unexplored, and the model cannot provide natural-language rationales.

Ethics Statement

The study uses preference data that includes safety-related content under a stated training purpose and reports implementation details to support reproducibility.

Data use: Safety-related data, including PKU-SafeRLHF, is used solely to train the reward model to distinguish safe from unsafe responses.MR2Bench-Image is constructed from user interactions on the AI2 Playground, with prompts summarized from user questions.
Reproducibility: The paper provides architecture and training details to facilitate reproduction of the results.Reported settings include value-head dimensions, activation, initialization, LoRA configuration, learning rate, epochs, batch size, and sequence length.

A.1 Ablation Study Results

The ablation study compares value-head architecture, response representation, and loss function, with the default configuration achieving the best average accuracy.

Ablation result: 64.8% average accuracy is achieved by the default MLP with SiLU, last-token pooling, and cross-entropy loss.All variants use Molmo2-4B with LoRA rank 64, learning rate 10^-4, three epochs, batch size 64, and a 73K training subset.

A.2 Baseline Evaluation Methodology

The evaluation compares single-pass multi-response scoring with independent discriminative scoring, pairwise generative judging, and direct best-of-4 selection under model-specific protocols.

Discriminative baselines: Independent discriminative reward models score each response separately, requiring N forward passes per sample.MM-RLHF-Reward additionally generates a critique, doubling the cost to 2N passes.
Generative baselines: Generative judges evaluate all N choose 2 response pairs and aggregate win counts as pseudo-scores.For four-response benchmarks, this requires six comparisons per sample.
Protocol details: Baseline model protocols vary in output format and video processing, including structured [[A]] or [[B]] verdicts and model-specific native pipelines.R1-Reward produces chain-of-thought analysis followed by a verdict, while LLaVA-Critic uniformly samples 16 video frames and has limited video performance because it was trained only on images.
Selection protocols: Direct best-of-N selection presents all responses simultaneously and chooses the best one, whereas pairwise aggregation selects by win counts.Table 6 compares these protocols on MR2Bench-Image and MR2Bench-Video.
Proposed method: The proposed model concatenates responses with separator tokens and extracts per-response scalar scores from a value head in one forward pass.This requires one forward pass per sample regardless of N.

A.3 Per-Category Benchmark Details

The appendix details benchmark configurations, computational comparisons, and controlled GRPO baselines for the multi-response reward model. Results emphasize lower computation, strong ranking quality, and greater stability than single-response reward optimization.

Computational efficiency: 2,937 TFLOPs for MR2Bench-Video and 1,468 TFLOPs for VideoRewardBench are theoretically estimated for MM-RLHF-Reward’s video FLOPs.The estimates compensate for vision-token expansion that FlopCounterMode does not fully capture.
Computational efficiency: 2−17× lower FLOPs than the most efficient baseline are achieved on image benchmarks, while remaining competitive on video benchmarks with a smaller 4B backbone.The model scores all N responses in a single forward pass, unlike per-response or pairwise baseline computation.
Benchmark evaluation: The Molmo2-4B reward model achieves the highest pairwise accuracy among discriminative reward models on MR2Bench-Video.Table 10 reports pairwise accuracy, Kendall’s τ, and best-of-N accuracy for the 4-response, 495-sample evaluation.
GRPO policy experiments: Two out of four single-response GRPO configurations exhibit reward hacking, whereas the comparative multi-response reward signal avoids this instability entirely.The LoRA-64 configuration repeats exclamation marks with VQAv2 ≈0%, while full fine-tuning raises hallucination to 52.1%.
Training data: 436K samples are selected from 881K raw samples across 10 source datasets, with 35% containing N>2 responses for listwise training.Source balancing uses square-root weighting and upsamples underrepresented categories such as reasoning, safety, and document understanding.
Benchmark construction: MR2Bench-Video uses approximately 94K pairwise judgments from 1,116 crowdworkers comparing responses from 19 models.The benchmark uses a balanced tournament design, with each model pair compared roughly 551 times across the question set.