Source-linked AI summary
Beyond Length Scaling: Synergizing Breadth and Depth for Generative Reward Models
Qiyuan Zhang, Yufei Wang, Tianhe Wu, Can Xu, Qingfeng Sun, Kai Zheng, Xue Liu, Chen Ma
TL;DR
Reward Models need reliable feedback for increasingly complex and diverse queries, while existing GRMs largely rely on unstructured CoT length scaling. Mix-GRM synthesizes breadth- and depth-oriented reasoning, trains them with SFT and RLVR, and finds task-dependent reasoning specialization with state-of-the-art performance across five benchmarks. The framework also exposes a scope boundary: its two-way reasoning taxonomy may be too coarse for hybrid tasks requiring simultaneous logical rigor and stylistic nuance.
Problem
Reward Models must provide reliable feedback for complex, diverse queries, but existing GRM methods predominantly scale unstructured CoT length despite task-dependent reasoning structures.
Method
Mix-GRM modularly reconfigures raw rationales into Breadth-CoT and Depth-CoT, then uses SFT and RLVR to align these mechanisms with task demands.
Results
Mix-GRM establishes state-of-the-art performance across five benchmarks; B-CoT favors subjective preference, D-CoT favors objective correctness, and RLVR sharpens this task-specific allocation.
Takeaways & Limitations
Reliable GRM reasoning depends on combining orthogonal mechanisms and calibrating them to task nature rather than merely expanding rationale length.
Takeaways & Limitations
The B-CoT/D-CoT dichotomy may be a coarse approximation for hybrid tasks requiring both deductive rigor and multidimensional nuance.
Abstract
from arXiv · showhide
Recent advancements in Generative Reward Models (GRMs) have demonstrated that scaling the length of Chain-of-Thought (CoT) reasoning considerably enhances the reliability of evaluation. However, current works predominantly rely on unstructured length scaling, ignoring the divergent efficacy of different reasoning mechanisms: Breadth-CoT (B-CoT, i.e., multi-dimensional principle coverage) and Depth-CoT (D-CoT, i.e., substantive judgment soundness). To address this, we introduce Mix-GRM, a framework that reconfigures raw rationales into structured B-CoT and D-CoT through a modular synthesis pipeline, subsequently employing Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable Rewards (RLVR) to internalize and optimize these mechanisms. Comprehensive experiments demonstrate that Mix-GRM establishes a new state-of-the-art across five benchmarks, surpassing leading open-source RMs by an average of 8.2\%. Our results reveal a clear divergence in reasoning: B-CoT benefits subjective preference tasks, whereas D-CoT excels in objective correctness tasks. Consequently, misaligning the reasoning mechanism with the task directly degrades performance. Furthermore, we demonstrate that RLVR acts as a switching amplifier, inducing an emergent polarization where the model spontaneously allocates its reasoning style to match task demands. The synthesized data and models are released at \href{https://huggingface.co/collections/DonJoey/mix-grm}{Hugging Face}, and the code is released at \href{https://github.com/Don-Joey/Mix-GRM}{Github}.
1 Introduction
Generative Reward Models improve evaluation by generating explicit rationales, but simply lengthening CoT overlooks task-dependent reasoning structures. Mix-GRM addresses this by synthesizing breadth- and depth-oriented reasoning and aligning them with task demands.
- GRMs generate explicit evaluation rationales before conclusions to transfer CoT’s generalization benefits to reward modeling.
- Existing GRM methods mainly scale CoT length and evaluation signals, although longer reasoning does not universally improve performance.
- Mix-GRM restructures raw rationales into Breadth-CoT and Depth-CoT within a unified reward-modeling framework.
- The framework synthesizes diverse principle coverage and judgment soundness, then pairs these mechanisms with subjective preference and objective correctness tasks.
- Across five benchmarks, Mix-GRM reaches state-of-the-art performance and downstream utility, while B-CoT favors preference, D-CoT favors correctness, and RLVR sharpens this allocation.
2 Related Work
Related work has expanded CoT beyond simple linear chains and has increasingly used longer rationales in Generative Reward Models. These approaches motivate structured reasoning designs for evaluation.
- Generative Reward Models extend LLM-as-a-Judge by generating natural-language rationales alongside preference decisions.
- Recent GRM work often uses reinforcement learning to elicit and stabilize longer CoT traces.
- CoT research has introduced branching and recurrent structures, including Tree of Thoughts and Graph of Thoughts, to search over partial thoughts.
3 Methodology
Mix-GRM standardizes raw rationales into modular units, synthesizes breadth- or depth-oriented reasoning, and trains a model to adapt its mechanism to task demands.
- Overview: Mix-GRM organizes evaluation into modular schema standardization, mechanism synthesis, and mechanism-adaptive alignment.The framework formalizes B-COT and D-COT within a unified reward-modeling pipeline.
- Problem Formulation: A normal GRM maps two candidate responses and an instruction to an evaluation rationale followed by a preference verdict.The full input is I = (x, yA, yB), and the output is (c, v).
- Modular Schema Standardization: Raw rationales are parsed into Principle–Judgment–Verdict units to make evaluation reasoning granular and verifiable.Each unit contains an evaluation principle, a judgment analyzing it, and a sub-verdict; K typically ranges from 3 to 5.
- Mechanism Synthesis: B-COT synthesizes diverse principles by sampling independent rationales, parsing them into schemas, and merging them into a non-redundant set.The resulting reasoning expands horizontal evaluative coverage for multifaceted preference judgments.
- Mechanism Synthesis: D-COT expands judgments through sequential verification, using a reasoning trace to ground refined judgments and favor deductive rigor.This mechanism trades horizontal coverage for substantive soundness in objective correctness tasks.
- Mechanism-Adaptive Alignment: Training applies SFT to mixed mechanism-specific CoT data, followed by GRPO-based RLVR that rewards verdict consistency with human labels.The reward-driven process induces the model to couple B-COT with preference tasks and D-COT with correctness tasks.
4 Experiment
The experiments evaluate Mix-GRM across reward benchmarks, mechanism efficiency, and downstream applications in offline RL and test-time scaling. Results show gains from mixture SFT, data efficiency, RLVR amplification, task-dependent reasoning, and downstream utility.
- Experimental Setup: Mix-GRM is evaluated on five general-purpose reward benchmarks covering math, coding, open-ended chat, factuality, and instruction-following.The study reports benchmark-level pairwise comparison accuracy and compares against discriminative and generative reward models.
- Overall Performance: 75.1 average score from mixture SFT surpasses RM-R1-Instruct by 5.0, DeepSeek-GRM-16B by 9.9, and RubricRM-8B by 0.4.This result is achieved without RL in the mixture-SFT stage.
- Overall Performance: 75.1 with 9K SFT samples is comparable to FARE-8B’s 75.9 using approximately 2.5M samples.The comparison is presented as evidence of higher training signal density from optimizing CoT mechanisms rather than expanding data alone.
- Overall Performance: RLVR improves Mix-GRM by 4.3 points, from 75.1 to 79.4, versus 3.6 points for Base-GRM, widening their gap from 1.8 to 2.5.The authors connect this stronger gain to mechanism allocation becoming polarized during RLVR.
- Mechanism Efficiency: B-COT improves preference but lowers correctness from 72.2 to 70.2, while D-COT raises deductive soundness but lowers preference from 68.2 to 65.9.Mix-GRM combines the orthogonal strengths, exceeding Base-GRM on average and reaching a 78.7 RLVR ceiling where single-mode mechanisms plateau.
- Downstream Utility: Mix-GRM yields a 12.1 instruction-following win rate after DPO and 46.4 math accuracy, while achieving the highest accuracy across four Best-of-10 reasoning and coding benchmarks.These downstream applications use Mix-GRM as a reward signal for DPO and as a verifier for candidate reranking.
5 Analysis
Analysis shows that reasoning structure, not length alone, determines task-specific reward-model behavior. RLVR expands this structure and enables the model to specialize its reasoning style, while breadth quality and compute remain important constraints.
- Switching CoT Mechanism Analysis: RLVR expands the mixed reasoning profile and induces task-specific polarization: D-COT for correctness and B-COT for preference.The model autonomously learns to amplify the mechanism aligned with each domain during reinforcement learning.
- Emergent Polarization Analysis: 73% structural matching after SFT improves during RL, indicating that specialization is learned rather than merely inherited from SFT priors.The analysis measures generated CoT structures using indicators including principle counts and trigger phrases.
- Scaling & Selection Analysis: 71.8/72.7 is achieved by BreadthTop10, outperforming BreadthFull and BreadthRand through consensus-based principle selection.BreadthTop10 selects the ten most frequent principles from four aggregated CoTs, suggesting that low-frequency principles add noise.
- Scaling & Selection Analysis: Performance improves monotonically as parallel CoTs increase from N = 1 to N = 4, confirming that aggregation expands evaluative coverage.The scaling result supports breadth as coverage expansion, but the selection ablation shows that volume alone is insufficient.
- Computational Overhead and Token Cost Analysis: B-COT and D-COT use approximately two reasoning passes, and proposed methods remain within the same order of magnitude of compute cost.The merge phase is restricted to merging and deduplication rather than adding information.
- Case Study: B-COT detects preference mismatches through lateral constraints, whereas D-COT catches subtle factual errors through step-by-step verification.The cases illustrate B-COT's coverage advantage and D-COT's logic-probing advantage across different evaluation demands.
6 Conclusion
The paper argues that reliable GRMs require structurally integrating complementary reasoning mechanisms rather than merely lengthening CoT. Mix-GRM aligns breadth and depth with task demands, shifting development toward structural optimization.
- 6 Conclusion: Mix-GRM integrates B-COT for multi-dimensional coverage with D-COT for judgment soundness through mechanism-adaptive alignment.The framework calibrates reasoning structure to the nature of the task.
Limitations
The framework has two stated limitations: its reasoning taxonomy is coarse, and RLVR-induced specialization may be rigid for hybrid tasks. Future work targets finer-grained and more flexible reasoning structures.
- Granularity of the Reasoning Manifold: The reasoning taxonomy captures subjective preference and objective correctness but may approximate a higher-dimensional alignment landscape too coarsely.Real-world tasks can lie on a continuous spectrum or combine deductive rigor with multidimensional nuances.
- Granularity of the Reasoning Manifold: More granular taxonomies are proposed to improve task-mechanism calibration.
- Rigidity in Explicit Hybrid Tasks: RLVR-induced convergence toward specialized reasoning poles may create structural rigidity in complex cross-domain scenarios.The limitation is especially relevant when tasks require hybrid reasoning demands.
- Rigidity in Explicit Hybrid Tasks: Future work proposes hybrid-slicing benchmarks and fine-grained structures such as soft routing to evaluate and support transitions across reasoning styles.The motivating applications combine rigorous deductive logic with stylistic writing.
A.2 Training Data Source Details
The training corpus is sampled across diverse alignment domains, while the document provides separate tables for reinforcement-learning hyperparameters and dataset composition.
- Training Configuration: Table 7 contains the hyperparameter settings for reinforcement learning.
- Training Data Sources: The corpus uses stratified random sampling to balance coverage across general chat, STEM, coding, math, safety, multilingual, and instruction-following domains.Source datasets, domains, and sampling statistics are reported in Table 8.
- Training Data Sources: Table 8 reports the composition and statistics of training data sampled from domain-specific sources.
A.3 Training Data Synthesis Details
CoT synthesis uses DeepSeek-v3 as the generator, with temperature-based sampling intended to diversify trajectories while preserving logical coherence.
- CoT Generation: DeepSeek-v3 using the 0324 snapshot serves as the backbone generator for synthesizing CoT data for SFT.
- CoT Generation: Sampling uses temperature T = 0.8 to promote trajectory diversity while maintaining logical coherence.
- Filtering: The synthesis process abstains from consistency filtering against the ground-truth human label.The passage states that empirical verification supports training on the full synthesized data, but the supplied text is truncated before the complete result.
A.4 Training Offline Reinforcement Learning Details
The offline reinforcement-learning setup controls temporal leakage, initializes a Llama-3-8B policy through SFT, and constructs DPO preferences by aggregating RM pairwise judgments over candidate responses.
- Policy Setup: Llama-3-8B is selected as the foundation model to control temporal data leakage and fairly compare with benchmark release dates.
- Policy Setup: The offline reinforcement-learning pipeline has two phases: SFT initialization followed by DPO.
- SFT Initialization: The SFT policy is fine-tuned on UltraChat and a random 40K-sample subset of MetaMathQA for 2 epochs at learning rate 2e−5.The maximum sequence length is 2,048 tokens.
- DPO Data Construction: DPO preference data combines 10K UltraFeedback instructions with 40K MetaMathQA instructions, generating N = 5 candidate responses per instruction.Candidate responses are produced with gpt-4o-mini at temperature 0.8.
- DPO Data Construction: For five candidates, the RM evaluates all (5 choose 2) = 10 response pairs and assigns +1 to the preferred response and 0 to the rejected response.
- DPO Data Construction: The highest cumulative score becomes the positive response and the lowest becomes the negative response for DPO training.
B.2 Benchmarks for Offline Reinforcement Learning Evaluation
The evaluation covers mathematical reasoning, instruction following, and Best-of-10 reranking across diverse benchmark subsets. The pipeline generates Vanilla-CoT, extracts modular schemas, and synthesizes target reasoning morphologies for fine-grained preference and correctness analysis.
- Mathematical Reasoning: Mathematical reasoning uses GSM8k, MATH, MAWPS, and TabMWP, spanning grade-school arithmetic, competition mathematics, and tabular processing.
- Instruction Following: Instruction following uses AlpacaEval-2 and Arena-Hard v0.1 with head-to-head auto-evaluation against baseline references and officially recommended configurations.
- Test-time Reranking: Best-of-10 reranking tests selection from mixed candidate pools on MATH, CHAMP, MBPP+, and BigCodeBench.
- Data Preparation: Vanilla-CoT uses MT-Bench and RewardBench prompts before extraction into the modular Principle–Judgment–Verdict schema and synthesis of target morphologies.
- Fine-grained Analysis: Detailed results separate subjective preference tasks in Table 10 from objective correctness tasks in Table 11 to decompose mechanism-level performance.
E Sensitivity and Robustness
The synthesis pipeline is evaluated for stability, noise robustness, and dependence on the underlying language model. Reported analyses indicate stable intermediate accuracy, negligible SFT sensitivity to rationale noise, and effectiveness with an open-weights model.
- Pipeline Stability: Pipeline accuracy remains highly stable across transformations; B-CoT merging resolves contradictions and improves accuracy, while D-CoT preserves correct reasoning paths.
- Robustness to Noise: SFT performance differs negligibly between 9K strictly verified CoT examples and 9K noisy CoT examples.
- Robustness to Noise: The method adopts unfiltered synthesized data to reduce curation cost without sacrificing performance, based on the noise ablation.
- Model Dependence: Using open-weights DEEPSEEK-V3 for schema extraction and raw rationale generation supports the pipeline’s robustness and generalizability across underlying LLM choice.
- Breadth-CoT Synthesis: Breadth-CoT synthesis merges semantically similar principles, refines their statements, curates supporting judgments, counts merged sources, and aggregates sub-verdicts by majority.