Source-linked AI summary
Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesis
Zicheng Kong, Dehua Ma, Zhenbo Xu, Alven Yang, Yiwei Ru, Haoran Wang, Zixuan Zhou, Fuqing Bie, Liuyu Xiang, Huijia Wu, Jian Zhao, Zhaofeng He
TL;DR
Multimodal reward models often have vision-centric coverage, opaque scalar outputs, and costly human-labeled supervision, limiting rubric-grounded evaluation across modalities. Omni-RRM combines automatically synthesized, teacher-reconciled preference data with structured rubric-grounded judgments and progressive SFT–GRPO training. It reports 70.4% five-benchmark Overall accuracy and a +17.0% relative gain over its backbone, with additional Best-of-N and text-only transfer results.
Problem
Existing multimodal reward models are often vision-centric, compress nuanced preferences into scalar scores, and depend on costly human annotation.
Source: §1 · p. 2, §2.2 · p. 4Method
Results
70.4% five-benchmark Overall accuracy and a +17.0% relative gain over its backbone; Omni-RRM also reports video accuracy of 80.2% on ShareGPT-Video and audio accuracy of 66.8% on Audio-HH and 65.0% on TA2T.
Source: §1 · pp. 2–3Takeaways & Limitations
Omni-RRM guides Best-of-N selection and exhibits positive transfer to text-only preference benchmarks.
Source: §1 · p. 3Structured rationales add reward-model decoding latency, although they deliver stronger Best-of-N selection gains on video and audio tasks.
Source: §4.4 · p. 12
Abstract
from arXiv · showhide
Multimodal large language models (MLLMs) struggle with alignment due to the limitations of existing reward models (RMs), which are predominantly vision-centric, dependent on costly human labels, and provide opaque scalar scores that fail to capture nuanced reasoning, leading to brittle alignment. We present Omni-RRM, an \textbf{Omni}-modal \textbf{R}ubric-grounded \textbf{R}eward \textbf{M}odel that generates multi-dimensional reward signals across text, image, video, and audio. To overcome the high cost and inherent inconsistency of human-centric evaluation in multi-dimensional reasoning, we introduce \textbf{Omni-Preference}, a high-quality dataset constructed via automatic rubric-grounded preference synthesis. In this pipeline, teacher models reconcile raw preferences into explicit justifications, ensuring that the synthesized supervision is both high-fidelity and interpretable. Omni-RRM is trained using a progressive SFT + GRPO regimen, specifically optimized to sharpen reward discrimination on low-margin, hard preference pairs. It achieves state-of-the-art accuracy on video (80.2\% on ShareGPT-Video) and audio benchmarks (66.8\% on Audio-HH-RLHF and 65.0\% on TA2T), yielding a five-benchmark Overall accuracy of 70.4\% and a +17.0\% relative gain over its backbone. Furthermore, Omni-RRM effectively guides Best-of-$N$ selection and exhibits robust transfer to text-only alignment. All resources, including the dataset, training and inference code, and model checkpoints are available at https://tmfk418.github.io/Omni-RRM.
1 Introduction
Existing multimodal reward models often offer vision-centric coverage, opaque scalar scores, and costly human-labeled supervision, while rubric-grounded signals are needed across modalities. Omni-RRM pairs automated, teacher-reconciled preference synthesis with structured rubric-grounded judgments and progressive training; it reports strong video and audio accuracy, 70.4% five-benchmark Overall accuracy, and a +17.0% relative gain over its backbone.
1 Introduction
- Existing multimodal reward models are often vision-centric, reduce nuanced preferences to opaque scalar scores, and rely on costly human annotation.The paper motivates automated high-fidelity preference supervision and unified, modality-aware rubric-grounded signals.Source: §1 · p. 2
- Omni-RRM generates structured rubric-grounded justifications before its final verdict, aiming to make reward judgments more auditable and criterion-level.The paper describes this structure as regularizing preference judgments toward human-auditable criteria.Source: §1 · p. 2
- 70.4% five-benchmark Overall accuracy and a +17.0% relative gain over its backbone; video and audio results include 80.2% on ShareGPT-Video, 66.8% on Audio-HH, and 65.0% on TA2T.The model also guides Best-of-N selection and transfers to text-only preference tasks.Source: §1 · pp. 2–3
- Omni-Preference uses automated preference synthesis and teacher-reconciled rationales, while progressive SFT–RL training sharpens discrimination on nuanced, low-margin pairs.The dataset is presented as dense, rubric-grounded supervision intended to support cross-modal reward calibration.Source: §1 · pp. 2–3
2 Related Work
The related work traces growing multimodal capabilities and alignment needs, then reviews reward models’ role in preference optimization and their coverage and interpretability gaps. Omni-RRM is positioned as combining a shared rubric, modality-aware evidence, automatic preference synthesis, and rubric-aware optimization.
2.1 Multi-modal Large Language Models
- Broader multimodal capabilities increase demand for alignment that generalizes across modalities, while heterogeneous inputs, modality-specific failures, and fragmented benchmarks complicate preference alignment.Reward models provide training-time supervision and inference-time scores for reranking or Best-of-N selection.Source: §2.1 · p. 3, §2.2 · p. 4
2.2 Preference Alignment
- Reward models instantiate preferences as learnable signals for post-training and inference-time reranking, but their reliability and coverage can bottleneck multimodal post-training.The related work attributes the challenge to heterogeneous inputs, modality-specific failure modes, and benchmark fragmentation.Source: §2.2 · p. 4
2.3 Reward Modeling
- Multimodal reward models remain imbalanced toward vision and often provide scalar scores or free-form critiques without fixed criterion-wise accountability.Recent rubric-grounded methods are mostly textcentric; Omni-RRM combines a shared five-criterion rubric, modality-aware instantiations, automatic synthesis, and rubric-aware GRPO across text, image, video, and audio.Source: §2.3 · p. 4
3 Method
Omni-RRM formulates reward modeling as structured generation of scores, a preference verdict, and criterion-level justifications under a shared five-criterion rubric. It builds Omni-Preference through teacher-reconciled preference synthesis and trains the model progressively with SFT followed by GRPO to refine low-margin discrimination.
3 Method
- Omni-RRM generates overall scores, a preference label, and five criterion-level comparative justifications rather than predicting preference as a scalar alone.The shared criteria are fluency/coherence, relevance, accuracy/completeness, reasoning quality, and safety/ethical alignment; the same schema is used for SFT targets and GRPO reward computation.Source: §3 · p. 5
- The five criteria are shared across modalities, while evidence for image, video, and audio judgments is instantiated using modality-specific cues.Video criteria additionally consider temporal ordering, action progression, and event consistency; audio criteria consider acoustic events, spoken content, and consistency between audio cues and text.Source: §3 · pp. 5–6
3.1 Omni-Preference: A Rubric-Grounded Dataset for Omni-Modal Reward Modeling
- Omni-Preference pairs responses sampled from stronger and weaker generators, then retains pairs only when two teachers agree on a non-tie verdict consistent with scores and rubric justifications.The dataset prioritizes supervision reliability and cross-modality coverage; pairs with reconciled score margin below 2 are defined as hard.Source: §3.1 · pp. 6–7
3.2 Training Omni-RRM: A Progressive SFT–RL Approach
- SFT teaches schema-compliant rubric-grounded outputs before GRPO samples groups of outputs and optimizes bounded rewards for format validity, preference correctness, and substantive justifications.The staged regimen is intended to stabilize the output interface while refining discrimination on low-margin pairs.Source: §3.2 · p. 8
4 Experiments
Omni-RRM is evaluated on image, video, and audio preference benchmarks against proprietary and open-source models. The experiments report strong benchmark performance, gains from rubric-grounded training, improved Best-of-N selection, and cross-modal reinforcement under joint supervision.
4.1 Benchmarks and Metrics
- The evaluation measures preference accuracy on image, video, and audio tasks, including ShareGPT-Video pairs filtered for clear human-score margins and two audio-conditioned benchmarks.Audio-HH-RLHF retains original human-written responses and labels after prompts are converted with TTS; it does not cover real-world acoustic noise or speaker variability.Source: §4.1 · p. 9
4.2 Baselines
- Baselines include proprietary and open-source MLLMs, multimodal reward models, and matched ablations of rationale supervision and rubric-aware RL.The matched variants use the same SFT+RL budget as Omni-RRM.Source: §4.2 · p. 9
4.3 Experimental Results
- Omni-RRM-7B reaches 70.4% five-benchmark Overall accuracy, closely matching Gemini-2.5-Pro (70.6%), and sets open-source reward-model highs on video and audio benchmarks.It scores 80.2% on ShareGPT-Video, 66.8% on Audio-HH-RLHF, 65.0% on TA2T, and 72.9% on MM-RewardBench.Source: Table 2 · p. 10, §4.3 · p. 10
- 68.1% vs. 71.8% Overall for 7B shows that removing rubric-aware RL lowers performance; the rationale-free variant also declines sharply.The ablations compare removal of five-criterion rationales during SFT+RL and removal of the rubric-related reward term during GRPO.Source: §4.3 · p. 11, Table 3 · p. 12
- 70.4% Overall accuracy with SFT+RL exceeds the SFT result of 62.6%, with larger gains on Hard than Easy pairs (+12.0 pp vs. +6.6 pp).Omni-RRM-7B also improves on the Qwen2.5-Omni-7B backbone by +17.0% relative overall accuracy.Source: §4.3 · p. 11
- On MM-RewardBench, Omni-RRM-7B scores 72.9% versus Gemini-2.5-Pro at 63.3%, illustrating that specialized reward models can outperform a much larger general-purpose MLLM.R1-Reward-7B scores 72.3% and UnifiedReward-Think-7B scores 71.4% on the same benchmark.Source: §4.3 · p. 11
4.4 Inference-Time Alignment via Best-of-N
- At N=10, Omni-RRM improves over Self-consistency by +1.9 on MMMU (54.5 →56.4), +1.7 on Video-MME (52.4 →54.1), and +2.6 on AVQA (77.7 →80.3).The fixed Qwen2.5-Omni-7B generator uses reward-model single-elimination tournaments for BoN; Omni-RRM’s structured rationales add decoding latency.Source: §4.4 · p. 12, Figure 2 · p. 13
4.5 Inter-Modality Dynamics in Omni-Modal Reward Modeling
- With the same 3B backbone and training budget, full image-video-audio supervision outperforms narrower modality mixtures, while removing any modality hurts performance across domains.Omni-modal training yields 65.1% on audio versus 63.8% for audio-only training.Source: §4.5 · pp. 13–15, Figure 3 · p. 14
5 Conclusion
The conclusion presents Omni-RRM as a rubric-grounded framework for multimodal reward modeling. It reports strong benchmark accuracy, cross-modal transfer, and utility for inference-time response selection.
5 Conclusion
- Omni-RRM combines automatic rubric-grounded preference synthesis and Omni-Preference with dense criterion-level supervision to address manual labeling’s scalability and consistency bottlenecks.The dataset contains teacher-reconciled rationales, and the model uses structured five-criterion justifications.Source: §5 · p. 15
- Omni-RRM reaches 70.4% five-benchmark Overall accuracy and achieves state-of-the-art accuracy among open-source reward models, closely matching strong proprietary MLLMs on challenging multimodal benchmarks.The conclusion describes its structured judgments as more auditable and less dependent on opaque scalar patterns or shortcut cues across image, video, and audio.Source: §5 · p. 15
- The analysis suggests unified rubric-grounded modeling can induce cross-modal transfer, with preference signals from heterogeneous modalities reinforcing a more consistent reward boundary.Source: §5 · p. 15
- Best-of-N evaluation supports Omni-RRM’s practical effectiveness as a plug-and-play tool for inference-time response selection.Source: §5 · p. 15