PaperlayerView exact revision on arXiv

Source-linked AI summary

UniRRM: Unified Reasoning Reward Models Across Languages and Evaluation Paradigms

Peng Lai, Yichao Du, Junchao Wu, Weibo Gao, Linan Yue, Longyue Wang, Weihua Luo, Derek F. Wong, Guanhua Chen

arXiv:2609.05910v1cs.CLcs.AI

TL;DR

Open-ended reward modeling is limited by costly proprietary judges, static criteria, fragmented evaluation paradigms, and weak multilingual support. The paper introduces MixReward and UniRRM, which combine multilingual pairwise/listwise data with staged reasoning and adaptive rubrics. UniRRM-8B and UniRRM-14B achieve strong results across evaluation paradigms and generalize to unseen point-wise tasks.

  • Problem

    Open-ended reward modeling lacks reliable objective verification and remains constrained by costly judges, static rubrics, fragmented paradigms, and limited multilingual support.

  • Method

    The paper builds MixReward and trains UniRRM with staged reasoning, adaptive rubric generation, and unified evaluation across languages and paradigms.

  • Results

    UniRRM-8B and UniRRM-14B achieve robust performance across pairwise and listwise benchmarks and generalize to point-wise tasks without direct point-wise supervision.

  • Takeaways & Limitations

    UniRRM provides a scalable and interpretable framework for multilingual, multi-paradigm reward modeling.

Abstract

from arXiv · show

Reinforcement learning (RL) excels on tasks with verifiable rewards, but in open-ended tasks, the reliability of reward models remains a key challenge. Existing solutions either depend on costly proprietary LLM-as-a-Judge systems or opaque scalar reward models that lack interpretability. Recent works on generative reward models offer a promising alternative, but they remain constrained by static evaluation criteria, fragmented evaluation paradigms, and limited multilingual support. To address these challenges, we introduce \textbf{MixReward}, a large-scale multilingual dataset spanning six domains and 103 languages, containing both pairwise and listwise data, and propose \textbf{UniRRM}, a unified reasoning reward model supporting multiple languages and evaluation paradigms. UniRRM uses a staged reasoning chain to dynamically generate task-generic and instruction-specific criteria, enabling fine-grained, input-adaptive judgments while maintaining consistency across languages. Experiments demonstrate that UniRRM-8B and UniRRM-14B achieve performance close to the state-of-the-art for models of comparable size across multiple benchmarks, and are effective for unseen evaluation paradigms. In addition, ablation studies validate the reliability and effectiveness of UniRRM.

1. Introduction

UniRRM addresses limitations in open-ended reward modeling by combining multilingual, multi-paradigm data with adaptive reasoning-based evaluation. Its 8B and 14B models show strong performance across pairwise, listwise, and point-wise settings.

  • Open-ended RL lacks objective verification, while proprietary LLM judges are costly and latent reward models can be opaque.Existing generative reward models offer an alternative but remain limited by static rubrics, fragmented evaluation paradigms, and English-centric coverage.
  • MixReward spans 103 languages and multiple domains with pairwise and listwise data for unified reward-model training.The dataset is constructed through a rigorous pipeline and supports UniRRM's multilingual and multi-paradigm scope.
  • UniRRM dynamically generates task-relevant rubrics and evaluates inputs across multiple languages and evaluation paradigms.The method combines adaptive rubric generation with a unified evaluation pipeline and SFT-plus-RL training.
  • UniRRM-8B and UniRRM-14B improve over their original backbones on pairwise tasks by 6.1 and 4.8 percentage points, respectively.Both models achieve robust performance among models of comparable size across pairwise and listwise benchmarks.
  • UniRRM generalizes effectively to point-wise evaluation despite not being trained on point-wise tasks.Additional experiments and ablations support the model's effectiveness across evaluation paradigms.

2. The MixReward Dataset

MixReward is a multilingual, multi-domain preference dataset assembled through staged curation, language expansion, verification, and format unification. The resulting resource combines pairwise and model-assisted listwise samples for reward-model training.

  • MixReward contains 64,528 examples spanning six domains and 103 languages.The dataset is built from multiple high-quality open-source community datasets.
  • The collection covers chat and instruction following, reasoning and mathematics, code, and safety-related evaluation data.Its source datasets include creative writing, instruction following, mathematical reasoning, programming, and safety alignment tasks.
  • Data Curation: Data curation filters semantically sparse high-resource samples while preserving low-resource linguistic diversity.InsTagger identifies samples with insufficient semantic constraints, and mathematical data receive additional difficulty-aware filtering.
  • Language Identification and Expansion: Language labels are assigned using FastText, large-scale reasoning models, and manual verification for uncertain cases.This staged process expands language coverage while resolving low-confidence classifications.
  • Data Verification: Verification retains translated samples only when verifier agreement preserves the chosen-versus-rejected preference ordering.With two verifiers, original English samples with agreement at most 0.25 are discarded, while translated samples require at least 0.75.
  • Data Assembly: The final dataset combines pairwise data with Gemini-2.5-Flash-generated listwise samples.All data begin in pairwise format, and a sampled subset is expanded into listwise evaluations using negative optimization.

3. Unified Reasoning Reward Model

UniRRM unifies multilingual reward-model evaluation across pairwise, listwise, and point-wise paradigms through a staged reasoning chain that generates adaptive rubrics and structured judgments. It combines supervised fine-tuning with reinforcement learning and composite rewards to improve evaluation reliability and output validity.

  • Framework overview: UniRRM addresses fragmented, English-centric reward modeling by supporting multilingual evaluation across pairwise, listwise, and point-wise paradigms.Its generative outputs provide pairwise or listwise decisions from the final judgment and point-wise scores from the score vector.
  • Structured evaluation: UniRRM produces textual reasoning, candidate scores, and a final highest-score decision as one sequential structured evaluation chain.This autoregressive design supports deriving pairwise or listwise outcomes from the decision and point-wise results from the scores.
  • Adaptive rubric generation: The model first analyzes task intent, risks, objectives, and constraints before generating task-generic and instruction-specific evaluation criteria.The resulting rubrics use a 1–5 scale and provide an input-adaptive evaluation standard rather than fixed dimensions.
  • Training pipeline: Training uses a two-stage SFT-and-RL pipeline, with SFT initializing structured generation from correctly judged oracle sequences and GRPO addressing format hallucinations and reasoning-score mismatch.The RL stage optimizes a composite reward covering format validity, outcome consistency, and adaptive-rubric quality.

4. Experiments

The experiments evaluate UniRRM across pairwise, pointwise, and listwise settings against diverse baselines, finding strong performance and effective zero-shot pointwise generalization. Ablations examine reward components and teacher-model capability.

  • Main results: UniRRM-14B achieves 0.885 on MM-Eval and 0.757 on JudgeBench, outperforming vanilla LLM judges and specialized scalar reward models.
  • Main results: UniRRM-14B achieves 0.791 on RWBench2, the best result among tested generative reward models for listwise ranking.
  • Main results: UniRRM-14B reaches 0.868 average accuracy, slightly exceeding its teacher model’s 0.865 despite having significantly fewer parameters.
  • Pointwise generalization: UniRRM-14B achieves 0.771 average accuracy on pointwise tasks, exceeding M-Prometheus-14B at 0.723 without direct pointwise training.
  • Reward ablation: Removing adaptive-rubric rewards reduces performance across benchmarks, while jointly removing adaptive-rubric and format rewards causes the largest drop.
  • Teacher-model ablation: Stronger teacher models improve overall pairwise accuracy from 0.809 to 0.863 by producing richer adaptive-rubric guidance.

5. Analyses

The analyses show that combining supervised fine-tuning with reinforcement learning yields the most robust training behavior, while multilingual reasoning and teacher capability affect evaluation quality.

  • Training recipes: Combining SFT and GRPO consistently outperforms either strategy alone across benchmarks, improving pairwise accuracy and robustness.SFT mainly stabilizes initialization, formatting, and reasoning paths; GRPO refines evaluative behavior.
  • Training recipes: RL alone remains behind SFT-based approaches, while SFT alone lacks fine-grained pairwise preference modeling without subsequent reinforcement learning.
  • Language reasoning: The language-reasoning experiments compare target-language reasoning during RL with standard UniRRM across RewardBench, MM-Eval, and JudgeBench.

6. Practical Feasibility in RL Training Loops

UniRRM is tested as a reward model inside practical RL training loops on GSM8K and Math-500, where it improves the trained policy over both a Skywork reward baseline and the base model.

  • Experimental setup: The experiment uses Qwen3-0.6B as the policy, UniRRM-8B as the reward model, and merged GSM8K and Math-500 training data.
  • Practical RL training: UniRRM-8B improves performance over the Skywork reward baseline by +3.2 on Math-500 and +3.3 on GSM8K.
  • Practical RL training: Compared with the base model, UniRRM-8B produces gains of +5.3 on Math-500 and +3.9 on GSM8K.

7. Conclusion

The paper concludes that UniRRM provides a multilingual, multi-paradigm reasoning reward model built on MixReward. Its experiments support strong performance, unseen pointwise generalization, and effective multilingual evaluation.

  • Conclusion: UniRRM supports robust evaluation across pairwise, listwise, and pointwise paradigms using staged reasoning and adaptive rubric generation.
  • Conclusion: MixReward supplies the large-scale multilingual foundation for UniRRM’s fine-grained, input-adaptive judgments.
  • Conclusion: The reported experiments indicate near-state-of-the-art performance, unseen pointwise generalization, and effective handling of multilingual inputs.

Impact Statement

The paper positions UniRRM and MixReward as tools for more interpretable, reliable, and multilingual reward-model evaluation. It also situates these contributions within limitations of existing reasoning and multilingual judge systems.

  • UniRRM and MixReward aim to improve the interpretability, reliability, and multilingual generalization of reward models.
  • Existing reasoning reward models remain limited by pair-wise training and predefined evaluation criteria, restricting their applicability.
  • Reward-model research remains dominated by English data, while multilingual judge systems fall short of rigorous reliability standards.
  • MixReward’s original pair-wise data shows chosen responses are slightly longer on average than rejected responses.

C.1. Detials of Benchmarks

The paper evaluates UniRRM against diverse benchmarks, baselines, and training configurations, while describing the data distributions and GRPO-based optimization setup. The benchmark suite spans pair-wise, multilingual, and multiple reward-model paradigms.

  • Benchmarks: JudgeBench evaluates LLM-as-a-Judge systems on challenging pair-wise response comparisons across knowledge, reasoning, mathematics, and coding.It contains 350 response pairs from GPT-4o and 270 from Claude-3.5-Sonnet.
  • Benchmarks: MM-Eval targets multilingual evaluation through five core subsets covering 18 languages and a Language Consistency subset covering 122 languages.
  • Baselines: The baselines include scalar, rubric-based, reasoning, multilingual, and list-wise reward models with differing training methods and evaluation capabilities.Examples include RubricRM, RM-R1, M-Prometheus, mR3, and RewardAnything.
  • Training: UniRRM training uses full fine-tuning in both SFT and RL stages, with RL optimized using GRPO’s group-relative reward normalization.The advantage term is derived from rewards normalized within groups of outputs for the same query.

E. Efficiency Analysis

UniRRM combines substantial reasoning-token workloads with favorable inference efficiency on identical 4×H100 setups. Its multilingual results show strong consistency overall, while complex preference comparison remains the main cross-language weakness.

  • Inference efficiency: UniRRM-8B and UniRRM-14B generate the longest completions, at 1381.5 and 1340.8 tokens, respectively, in the efficiency comparison.Their total token volumes are 7.54M and 7.42M over 2,985 samples.
  • Inference efficiency: 25,954.2 tok/s gives UniRRM-8B about 25% higher throughput than mR3-Qwen3-8B, with latency of 0.097s versus 0.085s.
  • Inference efficiency: UniRRM-14B reaches 20,327.0 tok/s, around 24% above mR3-Qwen3-14B, and uses 0.4055 GPU-hours versus 0.5576 for RM-R1-DeepSeek-32B.
  • Multilingual evaluation: UniRRM-14B shows negligible English-to-multilingual drops in Chat and Reasoning, but Chat Hard drops 8.8% from 0.807 to 0.719.Chat and Reasoning drops are 0.2% and 0.9%, while Safety drops 1.5%.
  • Multilingual evaluation: Scaling from 8B to 14B improves Chat Hard by 2.9%, Reasoning by 2.4%, and Safety by 2.1%, with Chat improving 0.2%.Chat Hard gains are positive across all 23 languages.
  • Multilingual evaluation: Non-Latin-script languages tend to perform lower in Chat Hard, although every language maintains a competitive overall average for UniRRM-14B.

H. Rationale for Not Using Spearman Correlation in Point-wise Evaluation

The paper avoids Spearman correlation for point-wise evaluation because score magnitudes may not be globally comparable across samples. It instead evaluates point-wise capability through pair-wise benchmarks with known preferences.

  • Spearman correlation assumes model scores have stable, globally comparable meanings across samples.
  • Context, difficulty, and implicit reference standards can make point-wise scores incomparable across samples, weakening Spearman-based evaluation.
  • Closed-source reference scores may contain systemic biases and preference leakage, causing correlation metrics to reflect spurious shared preferences.
  • The authors therefore use pair-wise benchmarks with known preferences to evaluate point-wise capability.

I. Training Dynamic

During RL training, UniRRM steadily improves reasoning consistency, rubric adherence, and total reward while maintaining stable output-format compliance. A medical case study further shows that its rubric prioritizes safety by penalizing misleading certainty more heavily than incomplete advice.

  • RL optimization: Reasoning consistency improves overall during RL training despite exploration-related fluctuations.The upward Consistency_Reward trend indicates increasingly coherent reasoning chains.
  • RL optimization: Rubric_Reward rises from approximately 2.8 to over 3.6, representing the training process’s strongest and smoothest improvement.The passage attributes this gain to better understanding and execution of core evaluation criteria.
  • RL optimization: Format_Reward remains within a narrow 0.97–0.99 range throughout training.This stability indicates that output-format compliance does not degrade while the model optimizes harder objectives.
  • RL optimization: Total_Reward improves consistently, reflecting convergence while balancing formatting, logical consistency, and rubric alignment.The training analysis presents this convergence as evidence that the optimization process avoids instability or performance collapse.
  • Safety-sensitive evaluation: The medical case study penalizes deterministic prevention claims and debated remedies more heavily than incomplete but cautiously worded advice.Response B is penalized for claiming masks guarantee prevention and for Vitamin C claims, whereas Response A uses probabilistic wording despite omitting vaccination.
  • Safety-sensitive evaluation: The evaluation rubric explicitly scores safety and accuracy, completeness, clarity, relevance, and discriminative handling of helpfulness–safety trade-offs.Its criteria distinguish safe but unhelpful refusals, helpful safe responses, and helpful but dangerous responses.
Loading 2609.05910v1…