PaperlayerView exact revision on arXiv

Source-linked AI summary

Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferences

Zhuoran Jin, Hongbang Yuan, Kejian Zhu, Jiachun Li, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao

arXiv:2510.23451v1cs.CLcs.AIcs.CV

TL;DR

Existing reward models have limited modality coverage and largely overlook free-form, user-specific preferences. Omni-Reward combines an omni-modal benchmark and preference dataset with discriminative and generative reward models, achieving strong benchmark performance, including a 20% accuracy gain over the base model on Omni-RewardBench.

  • Problem

    Reward models largely focus on text and image, while fixed preference data and benchmarks often overlook video, audio, and user-specific preferences.

    Source: §1 · p. 2

  • Method

    Omni-Reward combines a five-modality, nine-task benchmark, a dataset of general and instruction-tuning preferences, and discriminative and generative reward models.

    Source: §1 · p. 2, §7 · p. 11

  • Results

    Omni-Reward performs strongly on Omni-RewardBench and comparably to or better than state-of-the-art models on public RM benchmarks, with a 20% accuracy gain over the base model on Omni-RewardBench.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Omni-Reward provides a unified framework for omni-modal reward modeling with free-form user preferences.

    Source: §7 · p. 11

    Omni-RewardBench contains several thousand human-labeled pairs, which may be insufficient for evaluations involving millions of examples.

    Source: B LIMITATIONS · p. 22

Abstract

from arXiv · show

Reward models (RMs) play a critical role in aligning AI behaviors with human preferences, yet they face two fundamental challenges: (1) Modality Imbalance, where most RMs are mainly focused on text and image modalities, offering limited support for video, audio, and other modalities; and (2) Preference Rigidity, where training on fixed binary preference pairs fails to capture the complexity and diversity of personalized preferences. To address the above challenges, we propose Omni-Reward, a step toward generalist omni-modal reward modeling with support for free-form preferences, consisting of: (1) Evaluation: We introduce Omni-RewardBench, the first omni-modal RM benchmark with free-form preferences, covering nine tasks across five modalities including text, image, video, audio, and 3D; (2) Data: We construct Omni-RewardData, a multimodal preference dataset comprising 248K general preference pairs and 69K instruction-tuning pairs for training generalist omni-modal RMs; (3) Model: We propose Omni-RewardModel, which includes both discriminative and generative RMs, and achieves strong performance on Omni-RewardBench as well as other widely used reward modeling benchmarks.

1 INTRODUCTION

Omni-Reward addresses modality imbalance and rigid binary preferences with a free-form multimodal benchmark, dataset, and discriminative and generative reward models. The models report strong Omni-RewardBench results and competitive performance on public benchmarks.

1 INTRODUCTION

  • Existing reward models mainly focus on text and images and train on fixed binary preferences, limiting support for video and audio and diverse personalized preferences.Source: §1 · p. 2
  • Omni-RewardBench covers nine tasks across text, image, video, audio, and 3D, with 3,725 human-annotated preference pairs.Source: §1 · pp. 2–3
  • Omni-RewardModel-BT uses a Bradley–Terry objective, while Omni-RewardModel-R1 generates a textual critic alongside a scalar score using only 3% of the dataset.Source: §1 · pp. 2–3
  • On the w/ Ties setting, Claude 3.5 Sonnet achieves 66.54% accuracy and Gemma-3 27B achieves 65.12%, while purpose-built multimodal reward models lag.Source: §1 · pp. 2–3
  • Omni-RewardModel reports 73.68% accuracy w/o Ties and 65.36% accuracy w/ Ties, with comparable-or-better performance on public reward-model benchmarks.Source: §1 · p. 3
  • Omni-RewardData combines 248K general preference pairs with 69K instruction-tuning pairs containing free-form preference descriptions.Source: §1 · pp. 2–3

2 OMNI-REWARDBENCH

Omni-RewardBench evaluates reward models across nine tasks and five modalities using free-form evaluation criteria. Its samples compare two responses under a criterion, with settings that either exclude or allow ties.

2.2 DATASET COLLECTION

  • The benchmark covers nine tasks across image, video, audio, text, and 3D, with task pathways including 3D generation, image editing, image understanding, video understanding, and audio understanding.Source: §2 · p. 3, §2.2 · pp. 4–5

2.1 TASK DEFINITION AND SETTING

  • Samples pair two candidate responses with a free-form criterion and preferred response; evaluation tests strict preferences without ties or allows ties.Source: §2.1 · p. 3

2.2 DATASET COLLECTION

  • Its collection also includes text generation, image synthesis, video synthesis, and audio generation, spanning prompts and responses from multiple datasets and models.Source: §2.2 · pp. 3–5

2.3 CRITERIA AND PREFERENCE ANNOTATION

  • Three annotators independently label preferences against manually created criteria; quality filtering removes instances with invalid criteria or conflicting preferences.The final benchmark contains 3,725 preference data covering nine tasks across all modalities.Source: §2.3 · p. 5

3 OMNI-REWARDMODEL

Omni-RewardData combines general preference pairs with instruction-tuning examples for free-form preferences. Based on it, Omni-Reward introduces discriminative and generative reward models with distinct scoring processes.

3.1 OMNI-REWARDDATA CONSTRUCTION

  • Omni-RewardData contains 317K preference pairs across T2T, TI2T, T2I, and T2V, combining general and fine-grained preferences.The dataset comprises 248K general preference pairs and 69K newly collected instruction-tuning pairs.Source: §3.1 · p. 5, §3 · p. 5
  • The general-preference portion collects existing datasets for T2T, TI2T, T2I, and T2V, including binary preferences and judgments of visual quality and semantic alignment.Source: §3.1 · pp. 5–6
  • Instruction-tuning examples pair a free-form user preference with responses and a preference label; model-generated criteria are checked by three models for consistency.The in-house subsets include 24K T2T, 28K TI2T, and 17K T2I examples.Source: §3.1 · p. 6

3.2 DISCRIMINATIVE REWARD MODELING WITH BRADLEY-TERRY

  • Omni-RewardModel-BT learns scalar response scores with a Bradley–Terry loss, using user preferences and task criteria as system messages while freezing vision and audio encoders.Source: §3.2 · p. 6

3.3 GENERATIVE REWARD MODELING WITH REINFORCEMENT LEARNING

  • Omni-RewardModel-R1 generates a Chain-of-Thought explanation followed by a preference prediction, and GRPO training compares that prediction with the ground-truth preference.It is trained from scratch on 10K Omni-RewardData samples without distillation from larger models.Source: §3.3 · pp. 6–7

4 EXPERIMENTS

The experiments evaluate generative and specialized reward models on Omni-RewardBench under two tie settings, then test Omni-RewardModel on that benchmark and other multimodal RM benchmarks. Omni-RewardModel performs strongly on Omni-RewardBench and generalizes to other benchmarks.

4.1 BASELINE REWARD MODELS

  • The baselines include 30 generative RMs built on state-of-the-art MLLMs and specialized models spanning image generation, multimodal understanding, and image/video generation.The generative set includes 24 open-source and six proprietary models, ranging from 3B to 72B; GPT-4o-Audio-Preview is used for TA2T and T2A.Source: §4.1 · pp. 7–8

4.2 IMPLEMENTATION DETAILS

  • Evaluation reports accuracy both after excluding ties and with a tie option; generative RMs critique both responses before making a preference decision.For discriminative RMs, w/ Ties accuracy is the maximum three-class accuracy obtained by varying the tie threshold.Source: §4.2 · p. 8

4.3 EVALUATION RESULTS ON OMNI-REWARDBENCH

  • Current RMs remain limited: Claude 3.5 Sonnet reaches 66.54% accuracy with ties, and task performance varies by up to 28.37% across modalities.T2A, T23D, and TI2I perform notably worse; current systems primarily focus on text and image, while audio and 3D remain underexplored.Source: §4.3 · pp. 8–9
  • 73.68% accuracy without ties and 65.36% with ties; Omni-RewardModel-BT also achieves SOTA on TA2T and T2A, while Omni-RewardModel-R1 surpasses specialized RMs with explicit reasoning.Figure 3 compares open-source, closed-source, and proposed models across nine tasks under both tie settings.Source: §4.3 · p. 9, Figure 3 · p. 9

4.4 EVALUATION RESULTS ON GENERAL REWARD MODELING BENCHMARKS

5 ANALYSIS

The analysis finds that mixed multimodal training data improves cross-task generalization and that instruction-tuning data matters. It also reports strong performance correlations among related understanding and generation tasks.

5.1 IMPACT OF TRAINING DATA COMPOSITION

  • Training on mixed multimodal data yields significantly better cross-task generalization than single-modality training, while removing instruction-tuning data causes a clear performance drop.Single-modality training on T2T, TI2T, or T2I and T2V data produces only marginal improvements.Source: §5.1 · p. 10

5.2 CORRELATION OF PERFORMANCE ON DIFFERENT TASKS

  • Understanding-task performance correlations range from 0.8 to 0.9, while video, 3D, and image generation correlations are mostly 0.7 to 0.8.The authors interpret these patterns as shared performance patterns within understanding and generation tasks, suggesting cross-modal generalization potential.Source: §5.2 · pp. 10–11, Figure 4 · p. 10

6 RELATED WORK

Prior multimodal reward models and benchmarks have expanded beyond text, but remain distributed across modalities, tasks, or evaluation needs. The section identifies a missing unified framework for evaluating reward models against textual criteria across multimodal scenarios.

6.1 MULTIMODAL REWARD MODEL

  • Multimodal reward models address image and video generation or vision-language tasks, but remain inadequate for fully omni-modal scenarios.UnifiedReward is described as the first unified reward model for visual understanding and generation, while most earlier models focus on narrower tasks.Source: §6.1 · p. 11

6.2 REWARD MODEL EVALUATION

  • Existing benchmarks cover chat, reasoning, safety, image and video generation, and multimodal evaluation, but lack a unified framework for specific textual criteria across multimodal scenarios.The overview also notes fine-grained reward modeling work in text-to-text tasks and multimodal preference data for post-training alignment.Source: §6.2 · p. 11

7 CONCLUSION

Omni-Reward is a unified framework for omni-modal reward modeling with free-form user preferences, comprising a benchmark, preference dataset, and family of reward models.

7 CONCLUSION

  • Omni-Reward combines a five-modality, nine-task benchmark, multimodal preference data with general and instruction-tuning examples, and discriminative and generative reward models.The paper presents the framework as addressing modality imbalance and preference rigidity, and reports strong model performance.Source: §7 · p. 11

ETHICS STATEMENT

The authors conducted the annotations voluntarily and without external crowdsourcing or paid annotation platforms. They report institutional IRB approval and document protocols and quality-control measures.

  • All annotation tasks were conducted voluntarily by the paper’s authors, with no external crowdsourcing or paid annotation platforms.Source: ETHICS STATEMENT · p. 12
  • The authors document detailed annotation protocols and quality-control measures to safeguard research integrity and mitigate potential biases.Source: ETHICS STATEMENT · p. 12
  • The study reports no sensitive personal data or human subjects outside annotation, and states that internal annotation received institutional IRB approval.Source: ETHICS STATEMENT · p. 12

REPRODUCIBILITY STATEMENT

The paper documents implementation and data-processing details and says it will release its benchmark, dataset, and model. It also describes annotation procedures, reports limitations, and states that LLMs were used only for language polishing.

REPRODUCIBILITY STATEMENT

  • The authors describe architectures, training, evaluation, data processing, and annotation protocols, and say they will release Omni-RewardBench, Omni-RewardData, and Omni-RewardModel under CC BY-NC 4.0.Source: REPRODUCIBILITY STATEMENT · p. 12

A LLM USAGE STATEMENT

  • LLMs were used only for grammar checking and language polishing, not for research ideas, experiment design, methodology, analysis, or substantive research.Source: A LLM USAGE STATEMENT · p. 22

B LIMITATIONS

  • Limitations include benchmark scale and task granularity, single-turn preference data, preliminary reinforcement learning for Omni-RewardModel-R1, and omitted modalities.Source: B LIMITATIONS · p. 22

C BROADER IMPACTS

  • Preference pairs may contain offensive, inappropriate, or sensitive prompts and responses, so users are advised to apply their own ethical guidelines.Source: C BROADER IMPACTS · p. 22

1. Objective

  • The annotation objective is to identify and label interpretable dimensions on which one model response is preferred over another for a given task.Source: §1 · p. 23

2. Task Definition

  • Each instance provides a task description and two responses, which annotators compare using concrete dimensions such as relevance, coherence, or visual quality.Source: §2 · p. 23

3. Annotation Procedure

  • Annotators write one or more complete-sentence evaluation dimensions and label each as favoring Response A, Response B, or neither.Source: §3 · p. 23

E.1 ETHICS

  • The authors state that annotation was voluntary, participants were informed about the task and data use, and internal efforts received institutional IRB approval.Source: §E.1 · p. 25

E.2 QUALITY CONTROL

The quality-control pipeline removed 38% of samples across criteria and preference annotation. The remaining annotations had a reported Krippendorff’s alpha of 0.701, and explicit textual criteria were intended to reduce ambiguity.

E.2 QUALITY CONTROL

  • 38% of samples were removed across the two quality-control stages: 23% for vague or overly specific criteria and 15% for annotator disagreement.Source: §E.2 · p. 25
  • 0.701 Krippendorff’s alpha indicates substantial agreement among annotators on preference labels.Source: §E.2 · p. 25
  • The annotation task was performed by a small group of PhD students, with procedures intended to safeguard consistency and mitigate potential biases.Source: §E.2 · p. 25
  • Explicit, well-defined textual criteria are designed to reduce ambiguity and limit effects of cultural or individual interpretive variation.Source: §E.2 · p. 25

F.1 BENCHMARK COMPARISON

Omni-RewardBench covers nine tasks across five modalities and supports free-form criteria for directly evaluating reward models. Its rankings differ from VLRewardBench, while the dataset retains some modality imbalance and instruction generation uses multi-model verification.

F.1 BENCHMARK COMPARISON

  • Omni-RewardBench spans nine tasks across text, image, video, audio, and 3D, and uniquely supports free-form criteria for direct reward-model evaluation.The comparison with existing benchmarks is presented in Table 7.Source: §F.1 · p. 26, Table 7 · p. 26
  • 0.4572 Spearman correlation between ten models’ OmniRewardBench and VLRewardBench rankings indicates that the benchmarks produce differentiated rankings.Source: §F.1 · p. 26

F.2 OMNI-REWARDBENCH STATISTICS

  • Some modality imbalance in preference-pair distribution remains because collecting high-quality data across modalities is inherently difficult.Source: §F.2 · p. 26

F.3 OMNI-REWARDDATA STATISTICS

  • Instruction generation uses multi-model verification to mitigate potential errors and biases from GPT-4o, with filtering framed as a classification task.Source: §F.3 · p. 26

G IMPLEMENTATION DETAILS

The appendix gives implementation details and examines chain-of-thought reasoning, free-form criteria, and scoring strategies, with additional task examples and evaluation prompts.

G IMPLEMENTATION DETAILS

  • Omni-RewardModel-BT uses MiniCPM-o-2.6, while Omni-RewardModel-R1 uses Qwen2.5-VL-7B-Instruct; both are trained for two epochs.The BT model freezes its vision and audio encoders; the R1 model freezes its vision encoder. Experiments use four A100 80GB GPUs, and overall scores average performance across supported modalities.Source: G IMPLEMENTATION DETAILS · p. 28

H ADDITIONAL EXPERIMENTAL RESULTS

  • Table 10 reports ablation results on Omni-RewardBench under the w/o Tie setting.The supplied passage identifies the table’s scope but gives no cell values or comparison.Source: Table 10 · p. 29

I.1 EFFECT OF CHAIN-OF-THOUGHT REASONING

  • CoT helps weaker generative reward models, but yields little or no gain and may slightly reduce performance in stronger models.Without CoT, models generate a preference judgment directly; with CoT, they first generate a textual critic.Source: Figure 7 · p. 29, Figure 8 · p. 29, §I.1 · p. 30

I.2 EFFECT OF FREE-FORM CRITERIA

  • Free-form criteria are harder when they conflict with models’ inherent preferences: GPT-4o-mini drops 26.32% and Claude-3.5-Sonnet drops 18.50% in average accuracy.The authors divide examples into invariant and shifted groups based on whether inherent preferences agree with criteria-based preferences.Source: §I.2 · p. 30

I.3 EFFECT OF SCORING STRATEGY

  • Pairwise scoring significantly outperforms pointwise scoring on Omni-RewardBench.Pointwise scoring assigns each response a scalar score; pairwise scoring compares responses directly.Source: §I.3 · p. 30

J DATA EXAMPLES

K PROMPT TEMPLATES

  • Evaluation prompts ask models to follow the specified criteria, avoid response-order bias, and return a prescribed preference verdict.Some prompts permit a tie verdict; others request a decisive A-or-B choice.Source: K PROMPT TEMPLATES · pp. 44, 46
Loading 2510.23451v1…